Heart disease refers to several types of heart conditions and it is the number one cause of death worldwide. To prevent heart disease, we must first learn how to reliably detect it. The heart disease data used in this study has various measurements on patients health and cardiovascular statistics.
Source: UChicago Medicine
Source of data
This study used dataset from a study of heart disease that has been open to the public at the UCI Machine Learning Repository which is being maintained by the Center for Machine Learning and Intelligent Systems at the University of California, Irvine.
Import packages
We shall use different set of packages in R
- Data preparation and exploration
- Machine learning packages
- Table formating
#Import packages
data_exploration_packages <- c("tidyverse", "plotly", "openxlsx")
machine_learning_packages <- c("caret","MASS", "car", "kernlab","rpart","randomForest","class","ada", "rda","e1071", "nnet","ipred", "dbarts", "klaR", "glmnet", 'earth')
table_formating_packages <- c("knitr","kableExtra")
if(!require(install.load)){
install.packages("install.load")
}
install.load::install_load(c(data_exploration_packages, machine_learning_packages, table_formating_packages))
Load and prepare the dataset
As a first step we must load the dataset.
Overview of heart disease dataset
Heart disease data
|
patient_id
|
slope_of_peak_exercise_st_segment
|
thal
|
resting_blood_pressure
|
chest_pain_type
|
num_major_vessels
|
fasting_blood_sugar_gt_120_mg_per_dl
|
resting_ekg_results
|
serum_cholesterol_mg_per_dl
|
oldpeak_eq_st_depression
|
sex
|
age
|
max_heart_rate_achieved
|
exercise_induced_angina
|
heart_disease_present
|
|
0z64un
|
1
|
normal
|
128
|
2
|
0
|
0
|
2
|
308
|
0.0
|
1
|
45
|
170
|
0
|
0
|
|
ryoo3j
|
2
|
normal
|
110
|
3
|
0
|
0
|
0
|
214
|
1.6
|
0
|
54
|
158
|
0
|
0
|
|
yt1s1x
|
1
|
normal
|
125
|
4
|
3
|
0
|
2
|
304
|
0.0
|
1
|
77
|
162
|
1
|
1
|
|
l2xjde
|
1
|
reversible_defect
|
152
|
4
|
0
|
0
|
0
|
223
|
0.0
|
1
|
40
|
181
|
0
|
1
|
|
oyt4ek
|
3
|
reversible_defect
|
178
|
1
|
0
|
0
|
2
|
270
|
4.2
|
1
|
59
|
145
|
0
|
0
|
|
ldukkw
|
1
|
normal
|
130
|
3
|
0
|
0
|
0
|
180
|
0.0
|
1
|
42
|
150
|
0
|
0
|
|
2gbyh9
|
2
|
reversible_defect
|
150
|
4
|
2
|
0
|
2
|
258
|
2.6
|
0
|
60
|
157
|
0
|
1
|
|
daa9kp
|
2
|
fixed_defect
|
150
|
4
|
1
|
0
|
2
|
276
|
0.6
|
1
|
57
|
112
|
1
|
1
|
|
3nwy2n
|
3
|
reversible_defect
|
170
|
4
|
0
|
0
|
2
|
326
|
3.4
|
1
|
59
|
140
|
1
|
1
|
|
1r508r
|
2
|
normal
|
120
|
3
|
0
|
0
|
0
|
219
|
1.6
|
0
|
50
|
158
|
0
|
0
|
|
ldg4b9
|
2
|
normal
|
120
|
4
|
0
|
0
|
2
|
302
|
0.4
|
1
|
66
|
151
|
0
|
0
|
|
xc17yq
|
1
|
normal
|
140
|
4
|
0
|
0
|
0
|
226
|
0.0
|
1
|
42
|
178
|
0
|
0
|
|
mpggsq
|
1
|
normal
|
140
|
3
|
0
|
0
|
0
|
335
|
0.0
|
1
|
64
|
158
|
0
|
1
|
|
zlyac8
|
2
|
normal
|
138
|
4
|
0
|
0
|
2
|
236
|
0.2
|
0
|
45
|
152
|
1
|
0
|
|
f06u72
|
2
|
reversible_defect
|
120
|
1
|
0
|
0
|
0
|
231
|
3.8
|
1
|
38
|
182
|
1
|
1
|
|
2fv3rc
|
2
|
reversible_defect
|
144
|
4
|
0
|
0
|
2
|
200
|
0.9
|
1
|
50
|
126
|
1
|
1
|
|
qyrkxn
|
2
|
normal
|
130
|
2
|
0
|
0
|
2
|
234
|
0.6
|
0
|
45
|
175
|
0
|
0
|
|
237mql
|
1
|
reversible_defect
|
130
|
4
|
1
|
0
|
0
|
253
|
1.4
|
1
|
60
|
144
|
1
|
1
|
|
mc750a
|
1
|
normal
|
130
|
2
|
0
|
0
|
2
|
204
|
0.0
|
1
|
29
|
202
|
0
|
0
|
|
30v796
|
1
|
normal
|
136
|
2
|
2
|
1
|
2
|
319
|
0.0
|
0
|
58
|
152
|
0
|
1
|
|
cvux3j
|
1
|
normal
|
160
|
2
|
2
|
0
|
0
|
302
|
0.4
|
0
|
71
|
162
|
0
|
0
|
|
k8899q
|
1
|
reversible_defect
|
108
|
4
|
3
|
1
|
0
|
233
|
0.1
|
1
|
52
|
147
|
0
|
0
|
|
jhdvtb
|
1
|
normal
|
106
|
4
|
2
|
0
|
0
|
223
|
0.3
|
0
|
67
|
142
|
0
|
0
|
|
5g9v0h
|
1
|
fixed_defect
|
160
|
4
|
0
|
0
|
2
|
228
|
2.3
|
1
|
66
|
138
|
0
|
0
|
|
83asqd
|
1
|
normal
|
156
|
2
|
0
|
0
|
2
|
245
|
0.0
|
1
|
70
|
143
|
0
|
0
|
|
gla0im
|
2
|
normal
|
120
|
3
|
0
|
0
|
2
|
211
|
1.5
|
0
|
68
|
115
|
0
|
0
|
|
zzmfh7
|
1
|
normal
|
128
|
4
|
1
|
0
|
2
|
303
|
0.0
|
0
|
57
|
159
|
0
|
0
|
|
f4g1ay
|
1
|
normal
|
128
|
2
|
0
|
1
|
0
|
205
|
0.0
|
1
|
52
|
184
|
0
|
0
|
|
lek9q9
|
2
|
normal
|
140
|
3
|
0
|
0
|
2
|
185
|
3.0
|
1
|
60
|
155
|
0
|
1
|
|
8265rl
|
1
|
normal
|
110
|
3
|
0
|
0
|
0
|
175
|
0.6
|
1
|
51
|
123
|
0
|
0
|
|
6017a1
|
2
|
normal
|
130
|
3
|
0
|
0
|
2
|
214
|
2.0
|
1
|
41
|
168
|
0
|
0
|
|
z7xkou
|
2
|
reversible_defect
|
150
|
4
|
3
|
0
|
2
|
225
|
1.0
|
0
|
65
|
114
|
0
|
1
|
|
k7ef7h
|
3
|
reversible_defect
|
140
|
4
|
0
|
1
|
2
|
203
|
3.1
|
1
|
53
|
155
|
1
|
1
|
|
0n5fu0
|
1
|
normal
|
180
|
4
|
0
|
0
|
0
|
325
|
0.0
|
0
|
64
|
154
|
1
|
0
|
|
55xksg
|
2
|
reversible_defect
|
112
|
3
|
1
|
0
|
2
|
230
|
2.5
|
1
|
58
|
165
|
0
|
1
|
|
pjgqa3
|
1
|
normal
|
122
|
4
|
0
|
0
|
2
|
222
|
0.0
|
1
|
48
|
186
|
0
|
0
|
|
xkdz7j
|
1
|
reversible_defect
|
150
|
3
|
1
|
1
|
0
|
126
|
0.2
|
1
|
57
|
173
|
0
|
0
|
|
tpuevg
|
1
|
normal
|
124
|
4
|
0
|
0
|
0
|
209
|
0.0
|
0
|
62
|
163
|
0
|
0
|
|
ascl42
|
1
|
normal
|
120
|
2
|
1
|
0
|
2
|
269
|
0.2
|
0
|
74
|
121
|
1
|
0
|
|
1xwoe6
|
1
|
reversible_defect
|
128
|
4
|
1
|
0
|
0
|
255
|
0.0
|
1
|
52
|
161
|
1
|
1
|
|
ty4ik8
|
2
|
normal
|
150
|
3
|
0
|
1
|
0
|
243
|
1.0
|
1
|
61
|
137
|
1
|
0
|
|
gx6yxl
|
1
|
normal
|
135
|
3
|
0
|
0
|
2
|
252
|
0.0
|
0
|
63
|
172
|
0
|
0
|
|
hlmts5
|
1
|
normal
|
110
|
3
|
1
|
1
|
2
|
265
|
0.0
|
0
|
71
|
130
|
0
|
0
|
|
yx0q6k
|
1
|
normal
|
140
|
3
|
1
|
1
|
2
|
417
|
0.8
|
0
|
65
|
157
|
0
|
0
|
|
ep1o51
|
1
|
normal
|
108
|
3
|
0
|
0
|
2
|
267
|
0.0
|
0
|
54
|
167
|
0
|
0
|
|
gj1e5z
|
1
|
reversible_defect
|
124
|
2
|
0
|
0
|
0
|
261
|
0.3
|
1
|
57
|
141
|
0
|
1
|
|
6219kl
|
2
|
normal
|
125
|
3
|
0
|
1
|
2
|
245
|
2.4
|
1
|
51
|
166
|
0
|
0
|
|
rp9g6x
|
2
|
normal
|
112
|
4
|
0
|
0
|
0
|
149
|
1.6
|
0
|
71
|
125
|
0
|
0
|
|
1aeaff
|
2
|
reversible_defect
|
120
|
2
|
1
|
0
|
2
|
281
|
1.4
|
1
|
62
|
103
|
0
|
1
|
|
y3prof
|
1
|
normal
|
105
|
2
|
0
|
0
|
0
|
204
|
0.0
|
0
|
46
|
172
|
0
|
0
|
|
3drd48
|
2
|
reversible_defect
|
140
|
4
|
2
|
0
|
0
|
311
|
1.8
|
1
|
46
|
120
|
1
|
1
|
|
ejo7p3
|
1
|
normal
|
130
|
3
|
1
|
0
|
0
|
315
|
1.9
|
1
|
43
|
162
|
0
|
0
|
|
r7y4i1
|
1
|
reversible_defect
|
130
|
4
|
3
|
1
|
2
|
330
|
1.8
|
1
|
63
|
132
|
1
|
1
|
|
mznwxv
|
2
|
fixed_defect
|
130
|
3
|
1
|
1
|
2
|
256
|
0.6
|
1
|
56
|
142
|
1
|
1
|
|
27oevk
|
1
|
normal
|
130
|
4
|
0
|
0
|
2
|
330
|
0.0
|
0
|
61
|
169
|
0
|
1
|
|
jwqi3k
|
1
|
normal
|
130
|
3
|
0
|
0
|
0
|
233
|
0.4
|
1
|
44
|
179
|
1
|
0
|
|
328lkl
|
2
|
reversible_defect
|
110
|
4
|
1
|
0
|
0
|
239
|
2.8
|
1
|
54
|
126
|
1
|
1
|
|
tlk9o8
|
2
|
reversible_defect
|
120
|
4
|
2
|
0
|
0
|
267
|
1.8
|
1
|
62
|
99
|
1
|
1
|
|
aq2vrq
|
1
|
normal
|
120
|
2
|
0
|
0
|
0
|
295
|
0.0
|
1
|
42
|
162
|
0
|
0
|
|
ilogfb
|
1
|
normal
|
110
|
4
|
1
|
0
|
2
|
197
|
0.0
|
1
|
44
|
177
|
0
|
1
|
|
rv6siv
|
2
|
reversible_defect
|
115
|
3
|
0
|
0
|
2
|
564
|
1.6
|
0
|
67
|
160
|
0
|
0
|
|
m2a4i9
|
2
|
reversible_defect
|
130
|
4
|
0
|
0
|
0
|
305
|
1.2
|
0
|
51
|
142
|
1
|
1
|
|
pwigd8
|
3
|
reversible_defect
|
130
|
4
|
0
|
1
|
2
|
283
|
1.6
|
1
|
56
|
103
|
1
|
1
|
|
qwapdq
|
2
|
normal
|
112
|
2
|
0
|
0
|
0
|
160
|
0.0
|
0
|
45
|
138
|
0
|
0
|
|
4sd1xn
|
1
|
normal
|
110
|
4
|
0
|
0
|
2
|
254
|
0.0
|
0
|
50
|
159
|
0
|
0
|
|
nck22c
|
1
|
reversible_defect
|
126
|
4
|
0
|
0
|
2
|
282
|
0.0
|
1
|
35
|
156
|
1
|
1
|
|
m6zksp
|
2
|
normal
|
130
|
4
|
3
|
0
|
2
|
322
|
2.4
|
1
|
70
|
109
|
0
|
1
|
|
f70grj
|
2
|
normal
|
115
|
4
|
0
|
0
|
0
|
303
|
1.2
|
1
|
43
|
181
|
0
|
0
|
|
k1art8
|
2
|
normal
|
135
|
2
|
0
|
0
|
2
|
250
|
1.4
|
0
|
55
|
161
|
0
|
0
|
|
mcwqgs
|
2
|
reversible_defect
|
120
|
3
|
3
|
0
|
0
|
188
|
2.0
|
1
|
49
|
139
|
0
|
1
|
|
3jsjqk
|
1
|
normal
|
120
|
2
|
0
|
0
|
0
|
220
|
0.0
|
1
|
44
|
170
|
0
|
0
|
|
ik7hfs
|
1
|
normal
|
112
|
4
|
0
|
0
|
0
|
204
|
0.1
|
1
|
47
|
143
|
0
|
0
|
|
qwj1yf
|
1
|
reversible_defect
|
172
|
3
|
0
|
1
|
0
|
199
|
0.5
|
1
|
52
|
162
|
0
|
0
|
|
qvhk9e
|
1
|
normal
|
120
|
3
|
0
|
0
|
0
|
215
|
0.0
|
0
|
37
|
170
|
0
|
0
|
|
igwnqo
|
2
|
fixed_defect
|
126
|
3
|
1
|
1
|
0
|
218
|
2.2
|
1
|
59
|
134
|
0
|
1
|
|
4v0q7o
|
2
|
reversible_defect
|
178
|
4
|
2
|
1
|
0
|
228
|
1.0
|
0
|
66
|
165
|
1
|
1
|
|
hh2awp
|
2
|
normal
|
136
|
3
|
0
|
0
|
2
|
196
|
0.1
|
0
|
52
|
169
|
0
|
0
|
|
vfjppl
|
2
|
reversible_defect
|
124
|
4
|
1
|
0
|
2
|
266
|
2.2
|
1
|
54
|
109
|
1
|
1
|
|
6lu42b
|
2
|
reversible_defect
|
145
|
4
|
2
|
0
|
2
|
282
|
2.8
|
1
|
60
|
142
|
1
|
1
|
|
shiro4
|
2
|
reversible_defect
|
130
|
4
|
1
|
0
|
2
|
254
|
1.4
|
1
|
63
|
147
|
0
|
1
|
|
3wl3z4
|
3
|
normal
|
130
|
3
|
0
|
1
|
2
|
197
|
1.2
|
1
|
53
|
152
|
0
|
0
|
|
ebioez
|
2
|
reversible_defect
|
120
|
4
|
1
|
0
|
0
|
188
|
1.4
|
1
|
54
|
113
|
0
|
1
|
|
37c0vm
|
3
|
reversible_defect
|
110
|
2
|
0
|
0
|
0
|
229
|
1.0
|
1
|
48
|
168
|
0
|
1
|
|
v52zcs
|
2
|
reversible_defect
|
128
|
4
|
2
|
0
|
2
|
259
|
3.0
|
1
|
58
|
130
|
1
|
1
|
|
6nkcaw
|
1
|
normal
|
130
|
3
|
0
|
0
|
2
|
256
|
0.5
|
0
|
51
|
149
|
0
|
0
|
|
hfp05i
|
1
|
normal
|
118
|
3
|
3
|
0
|
2
|
149
|
0.8
|
1
|
49
|
126
|
0
|
1
|
|
grfxwd
|
1
|
normal
|
112
|
3
|
0
|
0
|
2
|
268
|
0.0
|
0
|
41
|
172
|
1
|
0
|
|
bvcxah
|
1
|
reversible_defect
|
140
|
4
|
1
|
0
|
0
|
177
|
0.0
|
1
|
59
|
162
|
1
|
1
|
|
i49srr
|
1
|
normal
|
150
|
3
|
0
|
0
|
0
|
168
|
1.6
|
1
|
57
|
174
|
0
|
0
|
|
93dbhq
|
1
|
normal
|
130
|
2
|
0
|
0
|
0
|
262
|
0.0
|
1
|
55
|
155
|
0
|
0
|
|
jscmp8
|
2
|
normal
|
134
|
2
|
0
|
0
|
0
|
271
|
0.0
|
0
|
49
|
162
|
0
|
0
|
|
zaytyf
|
2
|
normal
|
100
|
4
|
2
|
0
|
2
|
299
|
0.9
|
1
|
67
|
125
|
1
|
1
|
|
wze8qm
|
1
|
normal
|
135
|
3
|
0
|
1
|
0
|
304
|
0.0
|
0
|
54
|
170
|
0
|
0
|
|
w3933i
|
2
|
reversible_defect
|
140
|
4
|
2
|
0
|
2
|
293
|
1.2
|
1
|
60
|
170
|
0
|
1
|
|
7uch9x
|
2
|
normal
|
108
|
3
|
0
|
0
|
0
|
141
|
0.6
|
0
|
44
|
175
|
0
|
0
|
|
dy5hxt
|
1
|
reversible_defect
|
118
|
3
|
1
|
0
|
0
|
277
|
1.0
|
1
|
68
|
151
|
0
|
0
|
|
c0gkqc
|
2
|
fixed_defect
|
145
|
4
|
2
|
0
|
2
|
212
|
2.0
|
1
|
64
|
132
|
0
|
1
|
|
z5g5p3
|
2
|
normal
|
160
|
1
|
1
|
1
|
2
|
234
|
0.1
|
1
|
69
|
131
|
0
|
0
|
|
h3uzv8
|
1
|
normal
|
155
|
3
|
0
|
0
|
0
|
269
|
0.8
|
0
|
65
|
148
|
0
|
0
|
|
bthqr4
|
1
|
normal
|
150
|
1
|
0
|
1
|
2
|
283
|
1.0
|
0
|
58
|
162
|
0
|
0
|
|
rfj25e
|
1
|
normal
|
140
|
3
|
0
|
0
|
2
|
321
|
0.0
|
1
|
39
|
182
|
0
|
0
|
|
9f92et
|
2
|
normal
|
140
|
2
|
0
|
0
|
2
|
294
|
1.3
|
0
|
56
|
153
|
0
|
0
|
|
24fopx
|
2
|
reversible_defect
|
110
|
4
|
1
|
0
|
2
|
239
|
1.2
|
1
|
59
|
142
|
1
|
1
|
|
ldr1mz
|
1
|
normal
|
140
|
3
|
1
|
0
|
2
|
308
|
1.5
|
0
|
51
|
142
|
0
|
0
|
|
wokyol
|
1
|
reversible_defect
|
140
|
3
|
0
|
0
|
0
|
313
|
0.2
|
0
|
64
|
133
|
0
|
0
|
|
p5orwa
|
2
|
normal
|
130
|
4
|
2
|
0
|
0
|
303
|
2.0
|
0
|
64
|
122
|
0
|
0
|
|
s8dx1q
|
1
|
reversible_defect
|
150
|
3
|
0
|
0
|
2
|
232
|
1.6
|
1
|
54
|
165
|
0
|
0
|
|
7kf275
|
2
|
reversible_defect
|
160
|
4
|
1
|
0
|
2
|
289
|
0.8
|
1
|
55
|
145
|
1
|
1
|
|
e68djo
|
1
|
normal
|
125
|
1
|
1
|
0
|
2
|
213
|
1.4
|
1
|
51
|
125
|
1
|
0
|
|
3ze7pv
|
2
|
reversible_defect
|
124
|
4
|
0
|
0
|
2
|
274
|
0.5
|
1
|
48
|
166
|
0
|
1
|
|
0g192k
|
2
|
reversible_defect
|
128
|
4
|
1
|
0
|
0
|
263
|
0.2
|
1
|
64
|
105
|
1
|
0
|
|
3s141s
|
1
|
normal
|
120
|
2
|
0
|
0
|
0
|
244
|
1.1
|
0
|
50
|
162
|
0
|
0
|
|
6r9x2j
|
2
|
reversible_defect
|
140
|
4
|
3
|
0
|
0
|
298
|
4.2
|
1
|
51
|
122
|
1
|
1
|
|
sqddbc
|
2
|
reversible_defect
|
180
|
3
|
0
|
1
|
2
|
274
|
1.6
|
1
|
68
|
150
|
1
|
1
|
|
nizd9c
|
1
|
normal
|
112
|
3
|
0
|
0
|
0
|
250
|
0.0
|
1
|
41
|
179
|
0
|
0
|
|
lpub9d
|
1
|
normal
|
140
|
3
|
0
|
1
|
2
|
211
|
0.0
|
1
|
58
|
165
|
0
|
0
|
|
bv01fp
|
2
|
fixed_defect
|
135
|
2
|
0
|
0
|
0
|
203
|
0.0
|
1
|
41
|
132
|
0
|
0
|
|
9dqkpy
|
1
|
reversible_defect
|
110
|
4
|
0
|
0
|
2
|
172
|
0.0
|
1
|
41
|
158
|
0
|
1
|
|
2fqzg8
|
2
|
reversible_defect
|
132
|
4
|
1
|
0
|
0
|
353
|
1.2
|
1
|
55
|
132
|
1
|
1
|
|
1jruhz
|
2
|
normal
|
138
|
4
|
3
|
1
|
0
|
294
|
1.9
|
0
|
62
|
106
|
0
|
1
|
|
ju1wdc
|
2
|
normal
|
138
|
1
|
1
|
1
|
2
|
282
|
1.4
|
1
|
65
|
174
|
0
|
1
|
|
f4n8ny
|
1
|
normal
|
118
|
2
|
0
|
0
|
0
|
210
|
0.7
|
0
|
34
|
192
|
0
|
0
|
|
97v1yz
|
2
|
fixed_defect
|
140
|
4
|
0
|
0
|
0
|
192
|
0.4
|
1
|
57
|
148
|
0
|
0
|
|
6jcc1y
|
1
|
normal
|
130
|
3
|
3
|
1
|
2
|
246
|
0.0
|
1
|
53
|
173
|
0
|
0
|
|
tbo0wx
|
2
|
normal
|
160
|
4
|
3
|
0
|
2
|
286
|
1.5
|
1
|
67
|
108
|
1
|
1
|
|
4b32pd
|
1
|
normal
|
160
|
3
|
0
|
0
|
2
|
360
|
0.8
|
0
|
65
|
151
|
0
|
0
|
|
0ryxtv
|
2
|
normal
|
102
|
4
|
0
|
0
|
2
|
265
|
0.6
|
0
|
42
|
122
|
0
|
0
|
|
w1wgrq
|
2
|
reversible_defect
|
120
|
3
|
0
|
0
|
2
|
258
|
0.4
|
1
|
54
|
147
|
0
|
0
|
|
syvayq
|
3
|
reversible_defect
|
145
|
4
|
0
|
0
|
0
|
174
|
2.6
|
1
|
70
|
125
|
1
|
1
|
|
lq4ldx
|
3
|
normal
|
120
|
4
|
1
|
0
|
2
|
246
|
2.2
|
1
|
64
|
96
|
1
|
1
|
|
0zldrz
|
1
|
reversible_defect
|
94
|
3
|
1
|
0
|
0
|
227
|
0.0
|
1
|
51
|
154
|
1
|
0
|
|
98kb5h
|
2
|
normal
|
100
|
4
|
0
|
0
|
2
|
248
|
1.0
|
0
|
58
|
122
|
0
|
0
|
|
1wwvr0
|
1
|
normal
|
130
|
2
|
0
|
0
|
2
|
204
|
1.4
|
0
|
41
|
172
|
0
|
0
|
|
f1ziva
|
1
|
reversible_defect
|
132
|
3
|
2
|
0
|
2
|
224
|
3.2
|
1
|
58
|
173
|
0
|
1
|
|
e3dnw3
|
1
|
reversible_defect
|
125
|
4
|
2
|
0
|
2
|
300
|
0.0
|
1
|
58
|
171
|
0
|
1
|
|
08usun
|
1
|
reversible_defect
|
120
|
4
|
0
|
0
|
0
|
177
|
0.4
|
1
|
65
|
140
|
0
|
0
|
|
fzzkh7
|
1
|
reversible_defect
|
130
|
4
|
2
|
1
|
2
|
256
|
0.0
|
1
|
48
|
150
|
1
|
1
|
|
9upsjl
|
2
|
normal
|
150
|
4
|
0
|
0
|
0
|
244
|
1.4
|
0
|
62
|
154
|
1
|
1
|
|
ebloe5
|
1
|
normal
|
140
|
3
|
0
|
0
|
2
|
235
|
0.0
|
1
|
44
|
180
|
0
|
0
|
|
srm6ut
|
1
|
normal
|
130
|
2
|
0
|
0
|
2
|
219
|
0.0
|
1
|
44
|
188
|
0
|
0
|
|
noxsnw
|
3
|
reversible_defect
|
140
|
4
|
0
|
0
|
0
|
217
|
5.6
|
1
|
55
|
111
|
1
|
1
|
|
471q03
|
2
|
reversible_defect
|
120
|
1
|
0
|
0
|
2
|
193
|
1.9
|
1
|
56
|
162
|
0
|
0
|
|
uvwymz
|
1
|
reversible_defect
|
120
|
2
|
0
|
0
|
0
|
263
|
0.0
|
1
|
44
|
173
|
0
|
0
|
|
7eyvsi
|
2
|
reversible_defect
|
110
|
4
|
0
|
0
|
2
|
167
|
2.0
|
1
|
40
|
114
|
1
|
1
|
|
a946ij
|
2
|
reversible_defect
|
128
|
4
|
3
|
0
|
2
|
216
|
2.2
|
1
|
58
|
131
|
1
|
1
|
|
drdvf9
|
1
|
normal
|
140
|
2
|
2
|
0
|
0
|
195
|
0.0
|
0
|
63
|
179
|
0
|
0
|
|
z8yl4y
|
1
|
reversible_defect
|
140
|
1
|
0
|
0
|
0
|
199
|
1.4
|
1
|
40
|
178
|
1
|
0
|
|
mxabaz
|
2
|
normal
|
134
|
1
|
2
|
0
|
0
|
234
|
2.6
|
1
|
61
|
145
|
0
|
1
|
|
cpqg4x
|
1
|
normal
|
108
|
3
|
0
|
0
|
0
|
243
|
0.0
|
1
|
47
|
152
|
0
|
1
|
|
8c36yw
|
2
|
reversible_defect
|
128
|
3
|
1
|
0
|
2
|
229
|
0.4
|
1
|
57
|
150
|
0
|
1
|
|
x4yp0f
|
1
|
reversible_defect
|
108
|
2
|
0
|
0
|
0
|
309
|
0.0
|
1
|
54
|
156
|
0
|
0
|
|
9at0il
|
3
|
normal
|
125
|
3
|
1
|
0
|
2
|
273
|
0.5
|
1
|
54
|
152
|
0
|
0
|
|
nfag5b
|
2
|
reversible_defect
|
120
|
4
|
0
|
0
|
0
|
198
|
1.6
|
1
|
35
|
130
|
1
|
1
|
|
strmq8
|
1
|
normal
|
112
|
4
|
1
|
0
|
2
|
290
|
0.0
|
1
|
44
|
153
|
0
|
1
|
|
43k3gx
|
2
|
reversible_defect
|
130
|
3
|
1
|
0
|
0
|
263
|
1.2
|
0
|
62
|
97
|
0
|
1
|
|
fz84ac
|
1
|
normal
|
160
|
1
|
0
|
0
|
2
|
273
|
0.0
|
1
|
59
|
125
|
0
|
1
|
|
02cipp
|
1
|
normal
|
140
|
1
|
2
|
0
|
0
|
239
|
1.8
|
0
|
69
|
151
|
0
|
0
|
|
1ennzl
|
1
|
normal
|
130
|
3
|
0
|
0
|
0
|
275
|
0.2
|
0
|
48
|
139
|
0
|
0
|
|
isq8yp
|
1
|
normal
|
120
|
3
|
0
|
0
|
0
|
226
|
0.0
|
1
|
44
|
169
|
0
|
0
|
|
ewckbx
|
2
|
reversible_defect
|
130
|
4
|
2
|
0
|
2
|
206
|
2.4
|
1
|
60
|
132
|
1
|
1
|
|
dtljkq
|
1
|
normal
|
130
|
2
|
0
|
0
|
0
|
266
|
0.6
|
1
|
49
|
171
|
0
|
0
|
|
a2kf1z
|
1
|
reversible_defect
|
117
|
4
|
2
|
1
|
0
|
230
|
1.4
|
1
|
60
|
160
|
1
|
1
|
|
usnkhx
|
3
|
reversible_defect
|
160
|
4
|
3
|
0
|
2
|
164
|
6.2
|
0
|
62
|
145
|
0
|
1
|
|
hltlsl
|
2
|
reversible_defect
|
142
|
4
|
3
|
0
|
2
|
309
|
0.0
|
1
|
45
|
147
|
1
|
1
|
|
l0c19s
|
1
|
reversible_defect
|
142
|
4
|
0
|
0
|
2
|
226
|
0.0
|
1
|
53
|
111
|
1
|
0
|
|
lcexsf
|
1
|
normal
|
152
|
3
|
1
|
0
|
0
|
277
|
0.0
|
0
|
67
|
172
|
0
|
0
|
|
y3m2bd
|
1
|
reversible_defect
|
132
|
4
|
0
|
0
|
0
|
207
|
0.0
|
1
|
57
|
168
|
1
|
0
|
|
qcjf51
|
1
|
reversible_defect
|
120
|
4
|
0
|
0
|
2
|
249
|
0.8
|
1
|
46
|
144
|
0
|
1
|
|
7zbya5
|
3
|
fixed_defect
|
145
|
1
|
0
|
1
|
2
|
233
|
2.3
|
1
|
63
|
150
|
0
|
0
|
|
23gf0e
|
2
|
normal
|
110
|
1
|
0
|
0
|
2
|
211
|
1.8
|
1
|
64
|
144
|
1
|
0
|
|
qhz9ye
|
1
|
reversible_defect
|
150
|
4
|
0
|
0
|
2
|
270
|
0.8
|
1
|
58
|
111
|
1
|
1
|
|
u25507
|
1
|
normal
|
112
|
4
|
1
|
0
|
2
|
212
|
0.1
|
1
|
66
|
132
|
1
|
1
|
|
j9tw19
|
2
|
reversible_defect
|
118
|
4
|
0
|
0
|
0
|
219
|
1.2
|
1
|
39
|
140
|
0
|
1
|
|
5o32oi
|
1
|
reversible_defect
|
140
|
4
|
0
|
0
|
0
|
299
|
1.6
|
1
|
51
|
173
|
1
|
1
|
|
o63ri2
|
1
|
normal
|
140
|
4
|
0
|
0
|
0
|
239
|
1.2
|
1
|
54
|
160
|
0
|
0
|
|
5qfar3
|
2
|
reversible_defect
|
125
|
4
|
2
|
1
|
0
|
254
|
0.2
|
1
|
67
|
163
|
0
|
1
|
|
2s2b1f
|
2
|
normal
|
180
|
4
|
0
|
0
|
1
|
327
|
3.4
|
0
|
55
|
117
|
1
|
1
|
|
nsd00i
|
2
|
reversible_defect
|
125
|
3
|
0
|
0
|
0
|
309
|
1.8
|
1
|
64
|
131
|
1
|
1
|
|
0xw93k
|
1
|
normal
|
124
|
3
|
2
|
1
|
0
|
255
|
0.0
|
1
|
48
|
175
|
0
|
0
|
|
2nx10r
|
1
|
normal
|
160
|
3
|
1
|
0
|
0
|
201
|
0.0
|
0
|
54
|
163
|
0
|
0
|
Description of the dataset
There are 14 columns in the dataset, where the patient_id column is a unique and random identifier. The remaining 13 features are described below.
slope_of_peak_exercise_st_segment (type: int): the slope of the peak exercise ST segment, an electrocardiography read out indicating quality of blood flow to the heart
thal (type: categorical): results of thallium stress test measuring blood flow to the heart, with possible values normal, fixed_defect, reversible_defect
resting_blood_pressure (type: int): resting blood pressure
chest_pain_type (type: int): chest pain type (4 values)
num_major_vessels (type: int): number of major vessels (0-3) colored by flourosopy
fasting_blood_sugar_gt_120_mg_per_dl (type: binary): fasting blood sugar > 120 mg/dl
resting_ekg_results (type: int): resting electrocardiographic results (values 0,1,2)
serum_cholesterol_mg_per_dl (type: int): serum cholestoral in mg/dl
oldpeak_eq_st_depression (type: float): oldpeak = ST depression induced by exercise relative to rest, a measure of abnormality in electrocardiograms
sex (type: binary): 0: female, 1: male
age (type: int): age in years
max_heart_rate_achieved (type: int): maximum heart rate achieved (beats per minute)
exercise_induced_angina (type: binary): exercise-induced chest pain (0: False, 1: True)
Data preparation and preprocessing
Examine class label
As shown in the chart, we have 100 (56%) patients that have no heart disease while 80 (44%) of the patients have heart disease.
theme_set(theme_bw())# The current theme is automatically applied to every plot we draw
disease_data <- disease_data %>% mutate(heart_disease_present= fct_recode(heart_disease_present, absent = "0", present = "1"))
chart <- disease_data %>% count(heart_disease_present) %>% mutate(pct= round(n/sum(n)*100)) %>% ggplot(aes(x = heart_disease_present, y=n, fill= heart_disease_present)) + geom_bar(stat = 'identity', width = 0.4, show.legend = FALSE)+ labs(x='Heart disease present', y='Number of patients', caption = "Source: Heart disease data") +
scale_fill_manual(values = c( "present" = "red", "absent"= "green"), aesthetics = 'fill') +
geom_text(aes(label= str_c(pct, "%")), vjust= 4.5, size=2.5, colour='black') +
theme(legend.position = 'top', axis.title.y = element_text(size = 12, face='bold'), axis.title.x =element_text(size = 12, face='bold'), axis.text.x = element_text(angle = 50, vjust = 0.3, face = 'bold'))
ggplotly(chart, tooltip = c("x", "y"))
Visualize class separation by numeric features
The primary goal of visualization for classification problems is to understand which features are useful for class separation. In this section, we will start by visualizing the separation quality of numeric features.
# If you have column name as a character vector (e.g. col= 'age'), use .data[[col]]. If the column name or expression is supplied by the user, you can pass it to aes() or vars() using {{col}} i.e. curly curly syntax.(This is rlang syntax for writing a function)
plot_box = function(df, cols, col_x='heart_disease_present'){
for(col in cols){
p = ggplot(df, aes(x = .data[[col_x]], y = .data[[col]], fill = .data[[col_x]]))+ geom_boxplot(show.legend = FALSE) +
scale_fill_manual(values = c( "present" = "red", "absent" = "green"), aesthetics = 'fill') +
labs(x='Heart disease present', y= str_c(col),
title=str_c('Box plot of', col, '\n vs.',
col_x, sep = ' '), caption = "Source: Heart disease data") + theme(axis.text.x = element_text(face = 'bold'), axis.title.y = element_text(size = 12, face='bold'), axis.title.x =element_text(size = 12, face='bold'))
print(p)
}
}
num_cols = disease_data %>% select_if(is.numeric) %>% colnames()
plot_box(disease_data, num_cols)






Box plots are useful, since by construction we are forced to focus on the overlap (or not) of the quartiles of the distribution. In this case, we might ask the question like: is there sufficient differences in the quartiles for the feature to be useful in separation the label classes? There are two cases displayed above:
For age, max_heart_rate_achieved,oldpeak_eq_st_depression, num_major_vessels, there is useful separation between absent and present of heart disease patients. As one might expect, older people tends to have heart disease compared to the younger one.
On the other hand, resting_blood_pressure and serum_cholesterol_mg_per_dl does not seem to matter.
Visualizing class separation by categorical features
Now we will turn to the problem of visualizing the ability of categorical features to separate classes of the label. Ideally, a categorical feature will have very different counts of the categories for each of the label values. A good way to visualize these relationships is with bar plots. The code in the cell below creates side by side plots of the categorical variables for each of the labels categories.
# Since the facet_var will be supplied by the user, we pass curly curly syntax to vars() using {{facet_var}}
plot_bars = function(df, cat_cols, facet_var){
for(col in cat_cols){
p = ggplot(df, aes(x = .data[[col]], fill = .data[[col]])) +
geom_bar(show.legend = F) +
labs(x = col, y = "Number of patients", title = str_c('Bar plot of ', col, '\nfor heart disease')) +
facet_wrap(vars({{facet_var}}), scales = 'free_y') +
theme(axis.title.y = element_text(size = 12, face = 'bold'), axis.title.x = element_text(size = 12, face = 'bold'),
axis.text.x = element_text(angle = 90, hjust = 1, face = 'bold'))
print(p)
}
}
cat_cols <- disease_data %>% select_if(is.factor) %>% colnames()
cat_cols <- cat_cols[-8] # removing the class label
plot_bars(disease_data, cat_cols, heart_disease_present)







There is a lot of information in these plots. The key to interpretation of these plots is comparing the proportion of the categories for each of the label values. If these proportions are distinctly different for each label category, the feature is likely to be useful in separating the label.
There are several cases evident in these plots:
Some features such as slope_of_peak_exercise_st_segment, thal, and chest_pain_type have significantly different distribution of categories between the label categories.
Others features such as fasting_blood_sugar_gt_120_mg_per_dl, exercise_induced_angina and sex show small differences, but these differences are unlikely to be significant.
Other feature like resting_ekg_results has a dominant category with very few cases of other categories. These features will likely have very little power to separate the cases.
Notice that only a few of these categorical features will be useful in separating the cases.
Model building
In this secton we will perform two-class classification using binary classifiers. A classifier is a machine learning model that separates the label into categories or classes. In other words, classification models are supervised machine learning models which predict a categorical label. Common examples of classifiers include logistic regression, K Nearest Neighbour (KNN), Support Vector Machines (SVM), Random forest (RF), and Artificial neural network (NN). Some classifiers such as logistic regression, artificial neural network, gaussian process and random forest are known to predict probability of a given instance belonging to a particular class and are therefore called probabilistic classifiers. Classifiers of this nature use statistical inference to categorize the best label for a given instance. A predicted probability can then be converted into a class value by selecting the class label that has the highest probability. Unlike other algorithm such as K nearest neighbour which simply output the best class for a given instance. Classifiers such as logistic regression and adaboost were designed primarily for solving binary class problem and therefore will not work for multi-class problem.
In this case, our machine learning models used heart disease data to determine if a particular patient has heart disease or not. Thus, heart disease of patient is the classes we must predict.
Split the heart disease dataset into training and test datasets
we will create randomly sampled training and test data sets. The createDataPartition() function from the R caret package is used to create indices for the training data sample. In this case 80% of the data will be used for training the model. Since this data set is small, only 36 cases will be included in the test dataset. Execute this code and note the dimensions of the resulting data frame.
## The dimension of the training set is ( 144 14 )
## The dimension of test set is ( 36 14 )
Scale numeric features
Numeric features must be rescaled so they have a similar range of values. Rescaling prevents features from having an undue influence on model training simply because then have a larger range of numeric variables.
The code in the cell below uses the preProcess() function from the caret function. The processing is as follows:
The preprocessing model object is computed. In this case the processing includes centering and scaling the numeric feature. Notice that this model is fit only to the training data.
The scaling is applied to both the test and training partitions.
## Created from 144 samples and 14 variables
##
## Pre-processing:
## - centered (6)
## - ignored (8)
## - scaled (6)
One hot encoding
To convert all the nominal or factor variables to numeric, we need to create a dummy variable for each category of the categorical variables. Only one dummy variable is coded with a one for each set of categories. This is known as one hot encoding. By using numeric dummy variable, the entire training feature array is now numeric. To do this, we use dummyVars() function from the caret package. A predict method is applied to create numeric model matrices for training and test.
FALSE Dummy Variable Object
FALSE
FALSE Formula: ~.
FALSE <environment: 0x000000002d66b408>
FALSE 13 variables, 7 factors
FALSE Variables and levels will be separated by '.'
FALSE A full rank encoding is used
Feature selection
Feature selection can be an important part of model selection. In supervised learning, including features in a model which do not provide information on the label, is useless at best, and may prevent generalization at worst.
Feature selection can involve application of several methods. Two important methods include:
Eliminating features with low variance and zero variance. Zero variance features are comprised of the same values. Low variance features arise from features with most values the same and with few unique values. One way low variance features can arise, is from dummy variables for categories with very few members. The dummy variable will be mostly 0s with very few 1s.
Training machine learning models with features that are uninformative can create a variety of problems. An uninformative feature does not significantly improve model performance. In many cases, the noise in the uninformative features will increase the variance of the model predictions. In other words, uninformative models are likely to reduce the ability of the machine learning model to generalize.
FALSE freqRatio percentUnique zeroVar nzv
FALSE thal.fixed_defect 23 1.388889 FALSE TRUE
FALSE resting_ekg_results.1 143 1.388889 FALSE TRUE
We will remove the features that will not make our models to generalize well on the test and validation datasets.
Evaluation metric
The metric used for the evaluation of the performance of each model is logarithmic loss. In order to calculate Log Loss, the classifier must assign a probability to each class rather than simply yielding the most likely class. Mathematically Log Loss is defined as :
\[\text{Logloss}= - \frac{1}{n} \sum_{i=1}^n \left[ y_i \log(\hat{p}_i) + (1 - y_i) \log(1 - \hat{p}_i)\right]\]
where:
\(n\) - number of observations
\(log\) - the natural logarithm
\(y\) - a binary indicator (\(0\) or \(1\)) of whether class label \(c\) is the correct classification for observation \(o\)
\(p\) - the model’s predicted probability that observation \(o\) is of class \(c\).
Logarithmic loss provides a steep penalty for predictions that are both confident and wrong. That is, it takes into account the uncertainty of our model prediction based on how much it varies from the actual label. Logloss has no upper bound and it exists on the range \([0, \infty)\). Logloss nearer to \(0\) indicates higher accuracy, whereas if the logloss is away from \(0\) then it indicates lower accuracy. In general, the least logloss gives greater accuracy for the classifier. The goal is to minimize the logloss and a perfect classifier would have a logloss of precisely zero while less ideal classifiers have progressively larger values of logloss.
Computational section
The predictive performance of machine learning models depend on the structure of the dataset and proper data preparation will ensure the models work optimally. Since the best machine learning method on dataset cannot be known beforehand, in this section, we consider different catalogs of machine learning algorithms which included parametric and non parametric on heart disease data and we evaluated the performance of each model with logloss metric on test data.
R functions
We wrote two functions such as prob.prediction() and logloss().
prob.prediction(): This function takes yhat.model as input and that enables us to have a dataframe that comprises the probability of model class prediction and the label of the test set.
logloss(): The metric for evaluating performance of each classifier that was used in the heart disease data. This function as two input namely actual which is the true class label from the test set and predicted which is the probability of the classifier’s class prediction.
# Models to consider
# LDA model
lda.model<- train(y ~ ., data = xytrain, method = "lda", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC")
yhat.lda <- predict(lda.model, xtest, type = "prob")
lda_data <- prob.prediction(yhat.model = yhat.lda)
logloss_lda <- logloss(lda_data$ytest, lda_data$prob)
# GBM model
GBM.model <- train(y~., data=xytrain, method = "gbm", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary, seeds = vector(mode = "list", length = nrow(xytrain) + 1) %>% lapply(., function(x) 1:20)), metric = "ROC", tuneGrid = expand.grid(interaction.depth = 1:2, shrinkage = .1, n.trees = c(10, 50, 100), n.minobsinnode = 10),verbose = FALSE)
yhat.GBM <- predict(GBM.model, xtest, type = "prob")
GBM_data <- prob.prediction(yhat.model = yhat.GBM)
logloss_GBM <- logloss(GBM_data$ytest, GBM_data$prob)
# SVM model
svm.model <- train(y ~ ., data = xytrain, method = "svmLinear2", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE), tuneGrid = data.frame(cost = c(.25, .5, 1)))
yhat.svm <- predict(svm.model, xtest, type = "prob")
svm_data <- prob.prediction(yhat.model = yhat.svm)
logloss_svm <- logloss(svm_data$ytest, svm_data$prob)
# KNN model
knn.model <- train(y ~.,data=xytrain, method = "knn", trControl=trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC")
yhat.knn <- predict(knn.model, newdata = xtest, type="prob")
knn_data <- prob.prediction(yhat.model = yhat.knn)
logloss_knn <- logloss(knn_data$ytest, knn_data$prob)
# Ctree
ctree.model <- train(y ~ ., data = xytrain, method = "ctree", trControl = trainControl(method = "cv", number = 10, returnResamp = "all"))
yhat.ctree <- predict(ctree.model, newdata = xtest, type="prob")
ctree_data <- prob.prediction(yhat.model = yhat.ctree)
logloss_ctree <- logloss(ctree_data$ytest, ctree_data$prob)
# CART
cart.model <- train(y ~ ., data = xytrain, method = "rpart", trControl = trainControl(method = "cv", number = 10, returnResamp = "all"))
yhat.cart <- predict(cart.model, newdata = xtest, type="prob")
cart_data <- prob.prediction(yhat.model = yhat.cart)
logloss_cart <- logloss(cart_data$ytest, cart_data$prob)
# cforest
cforest.model <- train(y ~ ., data = xytrain, method = "cforest", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary, seeds = vector(mode = "list", length = nrow(xytrain) + 1) %>% lapply(., function(x) 1:20)), metric = "ROC", controls = party::cforest_unbiased(ntree = 20))
yhat.cforest <- predict(cforest.model, newdata = xtest, type="prob")
cforest_data <- prob.prediction(yhat.model = yhat.cforest)
logloss_cforest <- logloss(cforest_data$ytest, cforest_data$prob)
# gausspr model
gausspr.model <- gausspr(y~., data=xytrain)
FALSE Using automatic sigma estimation (sigest) for RBF or laplace kernel
yhat.gausspr <- predict(gausspr.model, xtest, type='prob')
gausspr_data <- prob.prediction(yhat.model = yhat.gausspr)
logloss_gausspr <- logloss(gausspr_data$ytest, gausspr_data$prob)
# rForest
rforest.model <- train(y ~ ., data = xytrain, method = "rf", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE,summaryFunction = twoClassSummary, seeds = vector(mode = "list", length = nrow(xytrain) + 1) %>% lapply(., function(x) 1:20)),metric = "ROC", ntree = 20,importance = TRUE)
yhat.rforest <- predict( rforest.model, xtest, type='prob')
rforest_data <- prob.prediction(yhat.model = yhat.rforest)
logloss_rforest <- logloss(rforest_data$ytest, rforest_data$prob)
# Adaboost
adaboost.model <- train(y ~ ., data = xytrain, method = "adaboost", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary, seeds =vector(mode = "list", length = nrow(xytrain) + 1) %>% lapply(., function(x) 1:20)),metric = "ROC")
yhat.adaboost <- predict( adaboost.model, xtest, type='prob')
adaboost_data <- prob.prediction(yhat.model = yhat.adaboost)
logloss_adaboost <- logloss(adaboost_data$ytest, adaboost_data$prob)
# Nnet
nnet.model <- train(y ~ ., data = xytrain, method = "nnet", trControl = trainControl(method = "cv", number = 10, returnResamp = "all"),
trace = FALSE)
yhat.nnet <- predict(nnet.model, xtest, type="prob")
nnet_data <- prob.prediction(yhat.model = yhat.nnet)
logloss_nnet <- logloss(nnet_data$ytest, nnet_data$prob)
# LogitBoost
logit.model <- train(y ~ ., data = xytrain, method = "LogitBoost", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC")
yhat.logit <- predict(logit.model, xtest, type="prob")
logit_data <- prob.prediction(yhat.model = yhat.logit)
logloss_logit <- logloss(logit_data$ytest, logit_data$prob)
# NaiveBayes
naiveBayes.model <- train( y ~ ., data = xytrain, method = "naive_bayes", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC")
yhat.naiveBayes <- predict(naiveBayes.model, xtest, type='prob')
naiveBayes_data <- prob.prediction(yhat.model = yhat.naiveBayes)
logloss_naiveBayes <- logloss(naiveBayes_data$ytest, naiveBayes_data$prob)
# MARS model
mars.model <- train(y~., data=xytrain, method='earth', trControl=trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE), tuneGrid = data.frame(degree = 1, nprune = (2:4)*2))
yhat.mars <- predict(mars.model, xtest, type='prob')
mars_data <- prob.prediction(yhat.model = yhat.mars)
logloss_mars <- logloss(mars_data$ytest, mars_data$prob)
# glmnet model
glmnet.model <- train(y~., data=xytrain, method='glmnet', trControl =trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary),metric = "ROC", tuneGrid = expand.grid(.alpha = seq(.05, 1, length = 15), .lambda = c((1:5)/10)))
yhat.glmnet <- predict(glmnet.model, xtest, type='prob')
glmnet_data <- prob.prediction(yhat.model = yhat.glmnet)
logloss_glmnet <- logloss(glmnet_data$ytest, glmnet_data$prob)
# xgbTree
xgbtree.model <- train(y ~ ., data = xytrain, method = "xgbTree", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC", tuneGrid = expand.grid(nrounds = c(1, 10), max_depth = c(1, 4), eta = c(.1, .4), gamma = 0, colsample_bytree = .7, min_child_weight = 1,subsample = c(.8, 1)))
yhat.xgbtree <- predict(xgbtree.model, xtest, type='prob')
xgbtree_data <- prob.prediction(yhat.model = yhat.xgbtree)
logloss_xgbtree <- logloss(xgbtree_data$ytest, xgbtree_data$prob)
j48.model <- train(y ~ ., data = xytrain, method = "J48", trControl = trainControl(method = "cv", number = 10, returnResamp = "all", classProbs = TRUE, summaryFunction = twoClassSummary), metric = "ROC")
yhat.j48 <- predict(j48.model, xtest, type='prob')
j48_data <- prob.prediction(yhat.model = yhat.j48)
logloss_j48 <- logloss(xgbtree_data$ytest, j48_data$prob)
# Models logloss comparision to check for the best model
evaluation <- tibble(lda= logloss_lda, GBM= logloss_GBM, SVM =logloss_svm, KNN = logloss_knn, ctree = logloss_ctree, CART = logloss_cart, cforest = logloss_cforest, `gauss process` = logloss_gausspr, `Random forest` = logloss_rforest, Adaboost = logloss_adaboost, `Neural network` = logloss_nnet, "Logistic" = logloss_logit, Naivebayes = logloss_naiveBayes, MARS = logloss_mars, glmnet = logloss_glmnet, XGBtree =logloss_xgbtree, j48=logloss_j48)
logloss= t(evaluation)
Model = rownames(logloss)
comparision_table= as_tibble(logloss) %>% add_column(Model) %>% rename(logloss="V1") %>% arrange(logloss) %>% add_column(SN= 1:length(Model),.after = 0) %>% dplyr::select(SN, Model, logloss)
Evaluation table
Models performance with logloss metric
|
SN
|
Model
|
logloss
|
|
1
|
glmnet
|
0.8130325
|
|
2
|
Adaboost
|
0.8239399
|
|
3
|
cforest
|
0.8649421
|
|
4
|
gauss process
|
0.8711982
|
|
5
|
XGBtree
|
0.9362978
|
|
6
|
SVM
|
0.9643937
|
|
7
|
MARS
|
0.9955285
|
|
8
|
ctree
|
1.0239794
|
|
9
|
Neural network
|
1.1142865
|
|
10
|
CART
|
1.1233441
|
|
11
|
GBM
|
1.3182116
|
|
12
|
lda
|
1.3716846
|
|
13
|
Logistic
|
1.7355291
|
|
14
|
Random forest
|
1.9371862
|
|
15
|
Naivebayes
|
3.3867352
|
|
16
|
j48
|
5.7350614
|
|
17
|
KNN
|
6.3656432
|
Optimal model
|
SN
|
Model
|
logloss
|
|
1
|
glmnet
|
0.8130325
|
This shows that glmnet is the optimal model.
Generalizing optimal model on validation dataset
We therefore, use the optimal model glmnet for predicting the class label on validation data.
# Validation dataset
validation_data <- read_csv("Data from DRIVENDATA/test_values.csv")
# Data wrangling and preprocessing of validation data
validation_data <- validation_data[-1] # patient_id column dropped
factor_variable_position <- c(1, 2, 4, 6, 7, 10, 13)
validation_data<- validation_data %>% mutate_at(vars(factor_variable_position),as_factor)
# Applying preprocessing to the validation dataset
validation_data <- predict(preProcess_scale_model,newdata = validation_data)
validation_dummy <- predict(dummies, newdata = validation_data)
# Convert to dataframe
xvalidation <- as_tibble(validation_dummy)
# Remove low variance columns on validation set
xvalidation <- xvalidation %>% dplyr::select(-c(resting_ekg_results.1,
thal.fixed_defect))
# Predicting the class label
validation_label <- predict(adaboost.model, xvalidation)
# Convert to data frame
validation_label <- tibble(heart_disease_present=validation_label)
validation_table <- validation_label %>% count(heart_disease_present) %>% mutate(Percent = round(n/sum(n)*100, 2))
Class prediction label
Summary of predicted labels on validation data
|
heart_disease_present
|
n
|
Percent
|
|
absent
|
48
|
53.33
|
|
present
|
42
|
46.67
|
## Parsed with column specification:
## cols(
## patient_id = col_character(),
## heart_disease_present = col_double()
## )
Validation data class prediction using optimal model (Adaboost classifier)
|
patient_id
|
heart_disease_present
|
|
olalu7
|
present
|
|
z9n6mx
|
absent
|
|
5k4413
|
present
|
|
mrg7q5
|
absent
|
|
uki4do
|
present
|
|
kev1sk
|
absent
|
|
9n6let
|
absent
|
|
jxmtyg
|
present
|
|
51s2ff
|
present
|
|
wi9mcs
|
absent
|
|
741h4l
|
absent
|
|
1ef64a
|
absent
|
|
wa2ix6
|
absent
|
|
8167zl
|
present
|
|
n6nldr
|
absent
|
|
ph85fp
|
absent
|
|
jfan5p
|
absent
|
|
7c4iz1
|
absent
|
|
ukigml
|
present
|
|
flwvnq
|
absent
|
|
5i4fw2
|
present
|
|
du1pqf
|
absent
|
|
vs68qz
|
absent
|
|
pfyez0
|
absent
|
|
azvkw2
|
present
|
|
cird1i
|
present
|
|
3bg32t
|
absent
|
|
xzd050
|
absent
|
|
eyi8et
|
present
|
|
ce4x2h
|
absent
|
|
sm91nr
|
present
|
|
2il8hh
|
absent
|
|
yq9cqg
|
present
|
|
520v5j
|
absent
|
|
ammgu2
|
present
|
|
jix8hj
|
absent
|
|
lj5zrq
|
present
|
|
16ceba
|
absent
|
|
93w44s
|
absent
|
|
bso17z
|
present
|
|
j2w2dc
|
present
|
|
74vwwl
|
absent
|
|
0z3fob
|
present
|
|
mr7zyz
|
absent
|
|
pp5n63
|
present
|
|
j0hix1
|
absent
|
|
rn209i
|
absent
|
|
nfit8e
|
present
|
|
nb73sy
|
present
|
|
i79t3w
|
present
|
|
9nv2d9
|
present
|
|
2xbeja
|
absent
|
|
lwg3wq
|
present
|
|
lrvqwb
|
absent
|
|
c6mepo
|
absent
|
|
6ued22
|
absent
|
|
112e9h
|
present
|
|
8jc7h2
|
absent
|
|
unykmj
|
absent
|
|
4yeztb
|
present
|
|
tgpy9u
|
absent
|
|
pf5wp6
|
present
|
|
cj8vj2
|
absent
|
|
9w6d9j
|
present
|
|
3l89wd
|
absent
|
|
83a6x1
|
present
|
|
oua0gr
|
present
|
|
j0hl96
|
present
|
|
dlkzyg
|
present
|
|
r0w4a8
|
absent
|
|
46dlca
|
absent
|
|
9fkefu
|
present
|
|
6uk6kl
|
present
|
|
c7olxr
|
present
|
|
iiyx0q
|
present
|
|
25vetx
|
present
|
|
073vc5
|
present
|
|
18abn0
|
absent
|
|
v5fsfs
|
absent
|
|
2ekoo7
|
absent
|
|
5bbknr
|
present
|
|
hr6pjx
|
absent
|
|
r4hsar
|
absent
|
|
4cezdf
|
absent
|
|
palhcc
|
present
|
|
bwoyg6
|
absent
|
|
j8i7ve
|
present
|
|
t2zn1n
|
absent
|
|
oxf8kj
|
present
|
|
aeiv0y
|
absent
|
Conclusion
This study considered \(17\) different catalogs of machine learning models which are carefully selected from the set of parametric and/or non-parametric models to choose the best optimal model with the least logloss. caret package was used to tune our different models and the optimal model glmnet predicted (48, 53.33%) has absent while (42, 46.67%) were predicted to have present of heart disease on the validation dataset.
If you like this writeup, you can also follow me on Twitter and Linkedin for more updates in R and Python for datascience.
LS0tDQp0aXRsZTogIk1hY2hpbmUgTGVhcm5pbmcgd2l0aCBhIEhlYXJ0Ig0Kc3VidGl0bGU6ICJQcmVkaWN0aW5nIEhlYXJ0IERpc2Vhc2UiDQphdXRob3I6IHwNCiAgfCBbT2d1bmRlcG8gRXpla2llbCBBZGViYXlvXShodHRwczovL2JpdC5seS9nYmdhbmFseXN0KQ0KICB8IFtJJ20gb24gVHdpdHRlcl0oaHR0cHM6Ly90d2l0dGVyLmNvbS9nYmdhbmFseXN0KQ0KDQpkYXRlOiAiYHIgZm9ybWF0KFN5cy50aW1lKCksICclQiAlZCwgJVknKWAiDQpvdXRwdXQ6IA0KICBodG1sX2RvY3VtZW50Og0KICAgIHRoZW1lOiB1bml0ZWQNCiAgICBoaWdobGlnaHQ6IGVzcHJlc3NvDQogICAgdG9jOiB0cnVlDQogICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlDQogICAgdG9jX2RlcHRoOiAzDQogICAgdG9jX2Zsb2F0OiB0cnVlDQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQ0KZWRpdG9yX29wdGlvbnM6IA0KICBjaHVua19vdXRwdXRfdHlwZTogY29uc29sZQ0KLS0tDQoNCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFKQ0KYGBgDQoNCkhlYXJ0IGRpc2Vhc2UgcmVmZXJzIHRvIHNldmVyYWwgdHlwZXMgb2YgaGVhcnQgY29uZGl0aW9ucyBhbmQgaXQgaXMgdGhlIFtudW1iZXIgb25lIGNhdXNlIG9mIGRlYXRoIHdvcmxkd2lkZV0od3d3LndvcmxkLWhlYXJ0LWZlZGVyYXRpb24ub3JnL3Jlc291cmNlcy9jYXJkaW92YXNjdWxhci1kaXNlYXNlcy1jdmRzLWdsb2JhbC1mYWN0cy1maWd1cmVzLykuIFRvIHByZXZlbnQgaGVhcnQgZGlzZWFzZSwgd2UgbXVzdCBmaXJzdCBsZWFybiBob3cgdG8gcmVsaWFibHkgZGV0ZWN0IGl0LiBUaGUgaGVhcnQgZGlzZWFzZSBkYXRhIHVzZWQgaW4gdGhpcyBzdHVkeSBoYXMgdmFyaW91cyBtZWFzdXJlbWVudHMgb24gcGF0aWVudHMgaGVhbHRoIGFuZCBjYXJkaW92YXNjdWxhciBzdGF0aXN0aWNzLg0KDQohW0hlYXJ0IGRpc2Vhc2VdKEltYWdlcy9IZWFydF9kaXNlYXNlLlBORykNCg0KU291cmNlOiBVQ2hpY2FnbyBNZWRpY2luZQ0KDQojIFNvdXJjZSBvZiBkYXRhDQoNClRoaXMgc3R1ZHkgdXNlZCBkYXRhc2V0IGZyb20gYSBzdHVkeSBvZiBoZWFydCBkaXNlYXNlIHRoYXQgaGFzIGJlZW4gb3BlbiB0byB0aGUgcHVibGljIGF0IHRoZSBbVUNJIE1hY2hpbmUgTGVhcm5pbmcgUmVwb3NpdG9yeV0oaHR0cDovL2FyY2hpdmUuaWNzLnVjaS5lZHUvbWwvaW5kZXgucGhwKSB3aGljaCBpcyBiZWluZyBtYWludGFpbmVkIGJ5IHRoZSBDZW50ZXIgZm9yIE1hY2hpbmUgTGVhcm5pbmcgYW5kIEludGVsbGlnZW50IFN5c3RlbXMgYXQgdGhlIFVuaXZlcnNpdHkgb2YgQ2FsaWZvcm5pYSwgSXJ2aW5lLg0KDQoNCiMjIEltcG9ydCBwYWNrYWdlcw0KDQpXZSBzaGFsbCB1c2UgZGlmZmVyZW50IHNldCBvZiBwYWNrYWdlcyBpbiBSDQoNCiogRGF0YSBwcmVwYXJhdGlvbiBhbmQgZXhwbG9yYXRpb24NCiogTWFjaGluZSBsZWFybmluZyBwYWNrYWdlcw0KKiBUYWJsZSBmb3JtYXRpbmcNCg0KYGBge3IgUl9saWJyYXJ5LCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLCBjb21tZW50PUZBTFNFLCBpbmNsdWRlPVRSVUV9DQoNCiNJbXBvcnQgcGFja2FnZXMNCg0KZGF0YV9leHBsb3JhdGlvbl9wYWNrYWdlcyA8LSBjKCJ0aWR5dmVyc2UiLCAicGxvdGx5IiwgIm9wZW54bHN4IikNCg0KbWFjaGluZV9sZWFybmluZ19wYWNrYWdlcyA8LSBjKCJjYXJldCIsIk1BU1MiLCAiY2FyIiwgImtlcm5sYWIiLCJycGFydCIsInJhbmRvbUZvcmVzdCIsImNsYXNzIiwiYWRhIiwgInJkYSIsImUxMDcxIiwgIm5uZXQiLCJpcHJlZCIsICJkYmFydHMiLCAia2xhUiIsICJnbG1uZXQiLCAnZWFydGgnKQ0KDQp0YWJsZV9mb3JtYXRpbmdfcGFja2FnZXMgPC0gYygia25pdHIiLCJrYWJsZUV4dHJhIikNCg0KDQppZighcmVxdWlyZShpbnN0YWxsLmxvYWQpKXsNCiAgaW5zdGFsbC5wYWNrYWdlcygiaW5zdGFsbC5sb2FkIikNCn0NCg0KaW5zdGFsbC5sb2FkOjppbnN0YWxsX2xvYWQoYyhkYXRhX2V4cGxvcmF0aW9uX3BhY2thZ2VzLCBtYWNoaW5lX2xlYXJuaW5nX3BhY2thZ2VzLCB0YWJsZV9mb3JtYXRpbmdfcGFja2FnZXMpKQ0KDQpgYGANCg0KDQojIyBMb2FkIGFuZCBwcmVwYXJlIHRoZSBkYXRhc2V0DQoNCkFzIGEgZmlyc3Qgc3RlcCB3ZSBtdXN0IGxvYWQgdGhlIGRhdGFzZXQuIA0KDQpgYGB7ciBUcmFpbmluZyBkYXRhc2V0LCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFLCBjb21tZW50PUZBTFNFLCBjYWNoZT1UUlVFLGluY2x1ZGU9VFJVRX0NCg0KIyBJbXBvcnQgZGF0YXNldA0KDQpkYXRhX3ZhbHVlcyA8LSByZWFkX2NzdigiRGF0YSBmcm9tIERSSVZFTkRBVEEvdHJhaW5fdmFsdWVzLmNzdiIpDQpkYXRhX2xhYmVscyA8LSByZWFkX2NzdigiRGF0YSBmcm9tIERSSVZFTkRBVEEvdHJhaW5fbGFiZWxzLmNzdiIpDQoNCiMgQ29uY2F0ZW5hdGluZyB0aGUgdHdvIGRhdGFzZXRzIA0KDQpkaXNlYXNlX2RhdGEgPC0gYmluZF9jb2xzKGRhdGFfdmFsdWVzLCBkYXRhX2xhYmVsc1ssMl0pDQoNCmBgYA0KDQoNCj4gKipPdmVydmlldyBvZiBoZWFydCBkaXNlYXNlIGRhdGFzZXQqKg0KDQpgYGB7ciBrYWJsZTEsIGV2YWw9VCwgaW5jbHVkZT1ULGVjaG89Rn0NCg0Ka2FibGUoZGlzZWFzZV9kYXRhLCBjYXB0aW9uID0gJ0hlYXJ0IGRpc2Vhc2UgZGF0YScsIGFsaWduID0gcmVwKCdjJywgMTUpKSAlPiUga2FibGVfc3R5bGluZyhib290c3RyYXBfb3B0aW9ucyA9IGMoInN0cmlwZWQiLCAiaG92ZXIiLCAiY29uZGVuc2VkIiwgInJlc3BvbnNpdmUiKSwgZml4ZWRfdGhlYWQgPSBUKSAlPiUgc2Nyb2xsX2JveCh3aWR0aCA9ICI5MDBweCIsIGhlaWdodCA9ICI1MDBweCIpDQogDQpgYGANCg0KDQojIyBEZXNjcmlwdGlvbiBvZiB0aGUgZGF0YXNldA0KDQpUaGVyZSBhcmUgMTQgY29sdW1ucyBpbiB0aGUgZGF0YXNldCwgd2hlcmUgdGhlIHBhdGllbnRfaWQgY29sdW1uIGlzIGEgdW5pcXVlIGFuZCByYW5kb20gaWRlbnRpZmllci4gVGhlIHJlbWFpbmluZyAxMyBmZWF0dXJlcyBhcmUgZGVzY3JpYmVkIGJlbG93Lg0KDQoqIHNsb3BlX29mX3BlYWtfZXhlcmNpc2Vfc3Rfc2VnbWVudCAodHlwZTogaW50KTogdGhlIHNsb3BlIG9mIHRoZSBwZWFrIGV4ZXJjaXNlIFNUIHNlZ21lbnQsIGFuIGVsZWN0cm9jYXJkaW9ncmFwaHkgcmVhZCBvdXQgaW5kaWNhdGluZyBxdWFsaXR5IG9mIGJsb29kIGZsb3cgdG8gdGhlIGhlYXJ0DQoNCiogdGhhbCAodHlwZTogY2F0ZWdvcmljYWwpOiByZXN1bHRzIG9mIHRoYWxsaXVtIHN0cmVzcyB0ZXN0IG1lYXN1cmluZyBibG9vZCBmbG93IHRvIHRoZSBoZWFydCwgd2l0aCBwb3NzaWJsZSB2YWx1ZXMgbm9ybWFsLCBmaXhlZF9kZWZlY3QsIHJldmVyc2libGVfZGVmZWN0DQoNCiogcmVzdGluZ19ibG9vZF9wcmVzc3VyZSAodHlwZTogaW50KTogcmVzdGluZyBibG9vZCBwcmVzc3VyZQ0KDQoqIGNoZXN0X3BhaW5fdHlwZSAodHlwZTogaW50KTogY2hlc3QgcGFpbiB0eXBlICg0IHZhbHVlcykNCg0KKiBudW1fbWFqb3JfdmVzc2VscyAodHlwZTogaW50KTogbnVtYmVyIG9mIG1ham9yIHZlc3NlbHMgKDAtMykgY29sb3JlZCBieSBmbG91cm9zb3B5DQoNCiogZmFzdGluZ19ibG9vZF9zdWdhcl9ndF8xMjBfbWdfcGVyX2RsICh0eXBlOiBiaW5hcnkpOiBmYXN0aW5nIGJsb29kIHN1Z2FyID4gMTIwIG1nL2RsDQoNCiogcmVzdGluZ19la2dfcmVzdWx0cyAodHlwZTogaW50KTogcmVzdGluZyBlbGVjdHJvY2FyZGlvZ3JhcGhpYyByZXN1bHRzICh2YWx1ZXMgMCwxLDIpDQoNCiogc2VydW1fY2hvbGVzdGVyb2xfbWdfcGVyX2RsICh0eXBlOiBpbnQpOiBzZXJ1bSBjaG9sZXN0b3JhbCBpbiBtZy9kbA0KDQoqIG9sZHBlYWtfZXFfc3RfZGVwcmVzc2lvbiAodHlwZTogZmxvYXQpOiBvbGRwZWFrID0gU1QgZGVwcmVzc2lvbiBpbmR1Y2VkIGJ5IGV4ZXJjaXNlIHJlbGF0aXZlIHRvIHJlc3QsIGEgbWVhc3VyZSBvZiBhYm5vcm1hbGl0eSBpbiBlbGVjdHJvY2FyZGlvZ3JhbXMNCg0KKiBzZXggKHR5cGU6IGJpbmFyeSk6IDA6IGZlbWFsZSwgMTogbWFsZQ0KDQoqIGFnZSAodHlwZTogaW50KTogYWdlIGluIHllYXJzDQoNCiogbWF4X2hlYXJ0X3JhdGVfYWNoaWV2ZWQgKHR5cGU6IGludCk6IG1heGltdW0gaGVhcnQgcmF0ZSBhY2hpZXZlZCAoYmVhdHMgcGVyIG1pbnV0ZSkNCg0KKiBleGVyY2lzZV9pbmR1Y2VkX2FuZ2luYSAodHlwZTogYmluYXJ5KTogZXhlcmNpc2UtaW5kdWNlZCBjaGVzdCBwYWluICgwOiBGYWxzZSwgMTogVHJ1ZSkNCg0KDQojIERhdGEgcHJlcGFyYXRpb24gYW5kIHByZXByb2Nlc3NpbmcNCg0KIyMgVHJhbnNmb3JtIGNvbHVtbiBkYXRhIHR5cGUNCg0KVGhlcmUgYXJlIGVpZ2h0IGNvbHVtbnMgaW4gdGhpcyBkYXRhc2V0IHdoaWNoIGRvIG5vdCBoYXZlIHRoZSBjb3JyZWN0IGRhdGEgdHlwZSBhcyBleHBlY3RlZC4gVGhpcyBpcyBhIGNvbW1vbiBzaXR1YXRpb24sIGFzIHRoZSBtZXRob2RzIHVzZWQgdG8gYXV0b21hdGljYWxseSBkZXRlcm1pbmUgZGF0YSB0eXBlIHdoZW4gbG9hZGluZyBmaWxlcyBjYW4gZmFpbCBzb21ldGltZXMuIFRoZSBjb2RlIGluIHRoZSBjZWxsIGJlbG93IGNvbnZlcnRlZCB0aGUgdmFyaWFibGVzIGludG8gdGhlIHJpZ2h0IGZvcm1hdCBpLmUuIGZhY3RvcnMuIA0KDQoNCmBgYHtyIGhlYXJ0IGRpc2Vhc2UgZGF0YXNldCwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1UUlVFLCBjb21tZW50PUZBTFNFLCBjYWNoZT1UUlVFLGluY2x1ZGU9VFJVRX0NCg0KZGlzZWFzZV9kYXRhIDwtICBkaXNlYXNlX2RhdGFbLTFdICMgIHBhdGllbnRfaWQgY29sdW1uIHdhcyBkcm9wcGVkDQoNCmZhY3Rvcl92YXJpYWJsZV9wb3NpdGlvbiA8LSBjKDEsIDIsIDQsIDYsIDcsIDEwLCAxMywgMTQpDQoNCmRpc2Vhc2VfZGF0YSA8LSBkaXNlYXNlX2RhdGEgJT4lIG11dGF0ZV9hdCh2YXJzKGZhY3Rvcl92YXJpYWJsZV9wb3NpdGlvbiksYXNfZmFjdG9yKQ0KDQpgYGANCg0KDQpUaGUgY2F0ZWdvcmljYWwgZmVhdHVyZXMgYXJlIG5vdyBjb2RlZCB3ZWxsLiBBZGRpdGlvbmFsbHksIHRoZSBsYWJlbCBpcyBub3cgY29kZWQgYXMgYSBiaW5hcnkgdmFyaWFibGUuIA0KDQojIyBFeGFtaW5lIGNsYXNzIGxhYmVsIA0KDQpBcyBzaG93biBpbiB0aGUgY2hhcnQsIHdlIGhhdmUgMTAwICg1NiUpIHBhdGllbnRzIHRoYXQgaGF2ZSBubyBoZWFydCBkaXNlYXNlIHdoaWxlIDgwICg0NCUpIG9mIHRoZSBwYXRpZW50cyBoYXZlIGhlYXJ0IGRpc2Vhc2UuIA0KDQpgYGB7ciBjaGFydCAxfQ0KDQp0aGVtZV9zZXQodGhlbWVfYncoKSkjIFRoZSBjdXJyZW50IHRoZW1lIGlzIGF1dG9tYXRpY2FsbHkgYXBwbGllZCB0byBldmVyeSBwbG90IHdlIGRyYXcNCg0KZGlzZWFzZV9kYXRhIDwtIGRpc2Vhc2VfZGF0YSAlPiUgbXV0YXRlKGhlYXJ0X2Rpc2Vhc2VfcHJlc2VudD0gZmN0X3JlY29kZShoZWFydF9kaXNlYXNlX3ByZXNlbnQsIGFic2VudCA9ICIwIiwgcHJlc2VudCA9ICIxIikpDQoNCmNoYXJ0IDwtIGRpc2Vhc2VfZGF0YSAlPiUgY291bnQoaGVhcnRfZGlzZWFzZV9wcmVzZW50KSAlPiUgbXV0YXRlKHBjdD0gcm91bmQobi9zdW0obikqMTAwKSkgJT4lIGdncGxvdChhZXMoeCA9IGhlYXJ0X2Rpc2Vhc2VfcHJlc2VudCwgeT1uLCBmaWxsPSBoZWFydF9kaXNlYXNlX3ByZXNlbnQpKSArIGdlb21fYmFyKHN0YXQgPSAnaWRlbnRpdHknLCB3aWR0aCA9IDAuNCwgc2hvdy5sZWdlbmQgPSBGQUxTRSkrIGxhYnMoeD0nSGVhcnQgZGlzZWFzZSBwcmVzZW50JywgeT0nTnVtYmVyIG9mIHBhdGllbnRzJywgY2FwdGlvbiA9ICJTb3VyY2U6IEhlYXJ0IGRpc2Vhc2UgZGF0YSIpICsNCiAgc2NhbGVfZmlsbF9tYW51YWwodmFsdWVzID0gYyggInByZXNlbnQiID0gInJlZCIsICJhYnNlbnQiPSAiZ3JlZW4iKSwgYWVzdGhldGljcyA9ICdmaWxsJykgKw0KICBnZW9tX3RleHQoYWVzKGxhYmVsPSBzdHJfYyhwY3QsICIlIikpLCB2anVzdD0gNC41LCBzaXplPTIuNSwgY29sb3VyPSdibGFjaycpICsNCnRoZW1lKGxlZ2VuZC5wb3NpdGlvbiA9ICd0b3AnLCBheGlzLnRpdGxlLnkgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDEyLCBmYWNlPSdib2xkJyksIGF4aXMudGl0bGUueCA9ZWxlbWVudF90ZXh0KHNpemUgPSAxMiwgZmFjZT0nYm9sZCcpLCBheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDUwLCB2anVzdCA9IDAuMywgZmFjZSA9ICdib2xkJykpDQoNCmdncGxvdGx5KGNoYXJ0LCB0b29sdGlwID0gYygieCIsICJ5IikpDQoNCmBgYA0KDQoNCiMjIFZpc3VhbGl6ZSBjbGFzcyBzZXBhcmF0aW9uIGJ5IG51bWVyaWMgZmVhdHVyZXMNCg0KVGhlIHByaW1hcnkgZ29hbCBvZiB2aXN1YWxpemF0aW9uIGZvciBjbGFzc2lmaWNhdGlvbiBwcm9ibGVtcyBpcyB0byB1bmRlcnN0YW5kIHdoaWNoIGZlYXR1cmVzIGFyZSB1c2VmdWwgZm9yIGNsYXNzIHNlcGFyYXRpb24uIEluIHRoaXMgc2VjdGlvbiwgd2Ugd2lsbCBzdGFydCBieSB2aXN1YWxpemluZyB0aGUgc2VwYXJhdGlvbiBxdWFsaXR5IG9mIG51bWVyaWMgZmVhdHVyZXMuIA0KDQpgYGB7ciBjaGFydDJ9DQoNCiMgSWYgeW91IGhhdmUgY29sdW1uIG5hbWUgYXMgYSBjaGFyYWN0ZXIgdmVjdG9yIChlLmcuIGNvbD0gJ2FnZScpLCB1c2UgLmRhdGFbW2NvbF1dLiBJZiB0aGUgY29sdW1uIG5hbWUgb3IgZXhwcmVzc2lvbiBpcyBzdXBwbGllZCBieSB0aGUgdXNlciwgeW91IGNhbiBwYXNzIGl0IHRvIGFlcygpIG9yIHZhcnMoKSB1c2luZyB7e2NvbH19IGkuZS4gY3VybHkgY3VybHkgc3ludGF4LihUaGlzIGlzIHJsYW5nIHN5bnRheCBmb3Igd3JpdGluZyBhIGZ1bmN0aW9uKQ0KDQpwbG90X2JveCA9IGZ1bmN0aW9uKGRmLCBjb2xzLCBjb2xfeD0naGVhcnRfZGlzZWFzZV9wcmVzZW50Jyl7DQogIGZvcihjb2wgaW4gY29scyl7DQogICAgcCA9IGdncGxvdChkZiwgYWVzKHggPSAuZGF0YVtbY29sX3hdXSwgeSA9IC5kYXRhW1tjb2xdXSwgZmlsbCA9IC5kYXRhW1tjb2xfeF1dKSkrIGdlb21fYm94cGxvdChzaG93LmxlZ2VuZCA9IEZBTFNFKSArDQogICAgICBzY2FsZV9maWxsX21hbnVhbCh2YWx1ZXMgPSBjKCAicHJlc2VudCIgPSAicmVkIiwgImFic2VudCIgPSAiZ3JlZW4iKSwgYWVzdGhldGljcyA9ICdmaWxsJykgKw0KICAgICAgbGFicyh4PSdIZWFydCBkaXNlYXNlIHByZXNlbnQnLCB5PSBzdHJfYyhjb2wpLA0KICAgICAgIHRpdGxlPXN0cl9jKCdCb3ggcGxvdCBvZicsIGNvbCwgJ1xuIHZzLicsDQogICAgICBjb2xfeCwgc2VwID0gJyAnKSwgY2FwdGlvbiA9ICJTb3VyY2U6IEhlYXJ0IGRpc2Vhc2UgZGF0YSIpICArIHRoZW1lKGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGZhY2UgPSAnYm9sZCcpLCBheGlzLnRpdGxlLnkgPSBlbGVtZW50X3RleHQoc2l6ZSA9IDEyLCBmYWNlPSdib2xkJyksIGF4aXMudGl0bGUueCA9ZWxlbWVudF90ZXh0KHNpemUgPSAxMiwgZmFjZT0nYm9sZCcpKQ0KICAgIA0KICBwcmludChwKSANCiAgfQ0KfQ0KDQpudW1fY29scyA9IGRpc2Vhc2VfZGF0YSAlPiUgc2VsZWN0X2lmKGlzLm51bWVyaWMpICU+JSBjb2xuYW1lcygpDQoNCnBsb3RfYm94KGRpc2Vhc2VfZGF0YSwgbnVtX2NvbHMpICAgIA0KYGBgDQoNCg0KQm94IHBsb3RzIGFyZSB1c2VmdWwsIHNpbmNlIGJ5IGNvbnN0cnVjdGlvbiB3ZSBhcmUgZm9yY2VkIHRvIGZvY3VzIG9uIHRoZSBvdmVybGFwIChvciBub3QpIG9mIHRoZSBxdWFydGlsZXMgb2YgdGhlIGRpc3RyaWJ1dGlvbi4gSW4gdGhpcyBjYXNlLCB3ZSBtaWdodCBhc2sgdGhlIHF1ZXN0aW9uIGxpa2U6IGlzIHRoZXJlIHN1ZmZpY2llbnQgZGlmZmVyZW5jZXMgaW4gdGhlIHF1YXJ0aWxlcyBmb3IgdGhlIGZlYXR1cmUgdG8gYmUgdXNlZnVsIGluIHNlcGFyYXRpb24gdGhlIGxhYmVsIGNsYXNzZXM/IFRoZXJlIGFyZSB0d28gY2FzZXMgZGlzcGxheWVkIGFib3ZlOg0KDQoxLiBGb3IgYWdlLCBtYXhfaGVhcnRfcmF0ZV9hY2hpZXZlZCxvbGRwZWFrX2VxX3N0X2RlcHJlc3Npb24sIG51bV9tYWpvcl92ZXNzZWxzLCB0aGVyZSBpcyB1c2VmdWwgc2VwYXJhdGlvbiBiZXR3ZWVuIGFic2VudCBhbmQgcHJlc2VudCBvZiBoZWFydCBkaXNlYXNlIHBhdGllbnRzLiBBcyBvbmUgbWlnaHQgZXhwZWN0LCBvbGRlciBwZW9wbGUgdGVuZHMgdG8gaGF2ZSBoZWFydCBkaXNlYXNlIGNvbXBhcmVkIHRvIHRoZSB5b3VuZ2VyIG9uZS4gDQoNCjIuIE9uIHRoZSBvdGhlciBoYW5kLCByZXN0aW5nX2Jsb29kX3ByZXNzdXJlIGFuZCBzZXJ1bV9jaG9sZXN0ZXJvbF9tZ19wZXJfZGwgZG9lcyBub3Qgc2VlbSB0byBtYXR0ZXIuIA0KDQojIyBWaXN1YWxpemluZyBjbGFzcyBzZXBhcmF0aW9uIGJ5IGNhdGVnb3JpY2FsIGZlYXR1cmVzDQoNCk5vdyB3ZSB3aWxsIHR1cm4gdG8gdGhlIHByb2JsZW0gb2YgdmlzdWFsaXppbmcgdGhlIGFiaWxpdHkgb2YgY2F0ZWdvcmljYWwgZmVhdHVyZXMgdG8gc2VwYXJhdGUgY2xhc3NlcyBvZiB0aGUgbGFiZWwuIElkZWFsbHksIGEgY2F0ZWdvcmljYWwgZmVhdHVyZSB3aWxsIGhhdmUgdmVyeSBkaWZmZXJlbnQgY291bnRzIG9mIHRoZSBjYXRlZ29yaWVzIGZvciBlYWNoIG9mIHRoZSBsYWJlbCB2YWx1ZXMuIEEgZ29vZCB3YXkgdG8gdmlzdWFsaXplIHRoZXNlIHJlbGF0aW9uc2hpcHMgaXMgd2l0aCBiYXIgcGxvdHMuIFRoZSBjb2RlIGluIHRoZSBjZWxsIGJlbG93IGNyZWF0ZXMgc2lkZSBieSBzaWRlIHBsb3RzIG9mIHRoZSBjYXRlZ29yaWNhbCB2YXJpYWJsZXMgZm9yIGVhY2ggb2YgdGhlIGxhYmVscyBjYXRlZ29yaWVzLg0KDQpgYGB7ciBjaGFydDN9DQoNCiMgU2luY2UgdGhlIGZhY2V0X3ZhciB3aWxsIGJlIHN1cHBsaWVkIGJ5IHRoZSB1c2VyLCB3ZSBwYXNzIGN1cmx5IGN1cmx5IHN5bnRheCB0byB2YXJzKCkgdXNpbmcge3tmYWNldF92YXJ9fQ0KDQpwbG90X2JhcnMgPSBmdW5jdGlvbihkZiwgY2F0X2NvbHMsIGZhY2V0X3Zhcil7DQogICBmb3IoY29sIGluIGNhdF9jb2xzKXsNCiAgICBwID0gZ2dwbG90KGRmLCBhZXMoeCA9IC5kYXRhW1tjb2xdXSwgZmlsbCA9IC5kYXRhW1tjb2xdXSkpICsgDQogICAgICBnZW9tX2JhcihzaG93LmxlZ2VuZCA9IEYpICArDQogICAgICBsYWJzKHggPSBjb2wsIHkgPSAiTnVtYmVyIG9mIHBhdGllbnRzIiwgdGl0bGUgPSBzdHJfYygnQmFyIHBsb3Qgb2YgJywgY29sLCAnXG5mb3IgaGVhcnQgZGlzZWFzZScpKSArDQogICAgICBmYWNldF93cmFwKHZhcnMoe3tmYWNldF92YXJ9fSksIHNjYWxlcyA9ICdmcmVlX3knKSArDQogICAgIHRoZW1lKGF4aXMudGl0bGUueSA9IGVsZW1lbnRfdGV4dChzaXplID0gMTIsIGZhY2UgPSAnYm9sZCcpLCBheGlzLnRpdGxlLnggPSBlbGVtZW50X3RleHQoc2l6ZSA9IDEyLCBmYWNlID0gJ2JvbGQnKSwNCiAgICAgICAgICAgIGF4aXMudGV4dC54ID0gZWxlbWVudF90ZXh0KGFuZ2xlID0gOTAsIGhqdXN0ID0gMSwgZmFjZSA9ICdib2xkJykpDQogICAgDQogICAgcHJpbnQocCkNCiAgICANCiAgfQ0KfQ0KDQpjYXRfY29scyA8LSBkaXNlYXNlX2RhdGEgJT4lIHNlbGVjdF9pZihpcy5mYWN0b3IpICU+JSBjb2xuYW1lcygpDQpjYXRfY29scyA8LSBjYXRfY29sc1stOF0gIyByZW1vdmluZyB0aGUgY2xhc3MgbGFiZWwNCg0KcGxvdF9iYXJzKGRpc2Vhc2VfZGF0YSwgY2F0X2NvbHMsIGhlYXJ0X2Rpc2Vhc2VfcHJlc2VudCkgICAgDQpgYGANCg0KDQpUaGVyZSBpcyBhIGxvdCBvZiBpbmZvcm1hdGlvbiBpbiB0aGVzZSBwbG90cy4gVGhlIGtleSB0byBpbnRlcnByZXRhdGlvbiBvZiB0aGVzZSBwbG90cyBpcyBjb21wYXJpbmcgdGhlIHByb3BvcnRpb24gb2YgdGhlIGNhdGVnb3JpZXMgZm9yIGVhY2ggb2YgdGhlIGxhYmVsIHZhbHVlcy4gSWYgdGhlc2UgcHJvcG9ydGlvbnMgYXJlIGRpc3RpbmN0bHkgZGlmZmVyZW50IGZvciBlYWNoIGxhYmVsIGNhdGVnb3J5LCB0aGUgZmVhdHVyZSBpcyBsaWtlbHkgdG8gYmUgdXNlZnVsIGluIHNlcGFyYXRpbmcgdGhlIGxhYmVsLiAgDQoNClRoZXJlIGFyZSBzZXZlcmFsIGNhc2VzIGV2aWRlbnQgaW4gdGhlc2UgcGxvdHM6DQoNCjEuIFNvbWUgZmVhdHVyZXMgc3VjaCBhcyBzbG9wZV9vZl9wZWFrX2V4ZXJjaXNlX3N0X3NlZ21lbnQsIHRoYWwsIGFuZCBjaGVzdF9wYWluX3R5cGUgaGF2ZSBzaWduaWZpY2FudGx5IGRpZmZlcmVudCBkaXN0cmlidXRpb24gb2YgY2F0ZWdvcmllcyBiZXR3ZWVuIHRoZSBsYWJlbCBjYXRlZ29yaWVzLg0KDQoyLiBPdGhlcnMgZmVhdHVyZXMgc3VjaCBhcyBmYXN0aW5nX2Jsb29kX3N1Z2FyX2d0XzEyMF9tZ19wZXJfZGwsIGV4ZXJjaXNlX2luZHVjZWRfYW5naW5hIGFuZCBzZXggIHNob3cgc21hbGwgZGlmZmVyZW5jZXMsIGJ1dCB0aGVzZSBkaWZmZXJlbmNlcyBhcmUgdW5saWtlbHkgdG8gYmUgc2lnbmlmaWNhbnQuDQoNCjMuIE90aGVyIGZlYXR1cmUgbGlrZSByZXN0aW5nX2VrZ19yZXN1bHRzIGhhcyBhIGRvbWluYW50IGNhdGVnb3J5IHdpdGggdmVyeSBmZXcgY2FzZXMgb2Ygb3RoZXIgY2F0ZWdvcmllcy4gVGhlc2UgZmVhdHVyZXMgd2lsbCBsaWtlbHkgaGF2ZSB2ZXJ5IGxpdHRsZSBwb3dlciB0byBzZXBhcmF0ZSB0aGUgY2FzZXMuDQoNCk5vdGljZSB0aGF0IG9ubHkgYSBmZXcgb2YgdGhlc2UgY2F0ZWdvcmljYWwgZmVhdHVyZXMgd2lsbCBiZSB1c2VmdWwgaW4gc2VwYXJhdGluZyB0aGUgY2FzZXMuDQoNCiMgTW9kZWwgYnVpbGRpbmcNCg0KSW4gdGhpcyBzZWN0b24gd2Ugd2lsbCBwZXJmb3JtICoqdHdvLWNsYXNzIGNsYXNzaWZpY2F0aW9uKiogdXNpbmcgKipiaW5hcnkgY2xhc3NpZmllcnMqKi4gQSBjbGFzc2lmaWVyIGlzIGEgbWFjaGluZSBsZWFybmluZyBtb2RlbCB0aGF0IHNlcGFyYXRlcyB0aGUgKipsYWJlbCoqIGludG8gY2F0ZWdvcmllcyBvciAqKmNsYXNzZXMqKi4gSW4gb3RoZXIgd29yZHMsIGNsYXNzaWZpY2F0aW9uIG1vZGVscyBhcmUgKipzdXBlcnZpc2VkKiogbWFjaGluZSBsZWFybmluZyBtb2RlbHMgd2hpY2ggcHJlZGljdCBhIGNhdGVnb3JpY2FsIGxhYmVsLiBDb21tb24gZXhhbXBsZXMgb2YgY2xhc3NpZmllcnMgaW5jbHVkZSBsb2dpc3RpYyByZWdyZXNzaW9uLCBLIE5lYXJlc3QgTmVpZ2hib3VyIChLTk4pLCBTdXBwb3J0IFZlY3RvciBNYWNoaW5lcyAoU1ZNKSwgUmFuZG9tIGZvcmVzdCAoUkYpLCBhbmQgQXJ0aWZpY2lhbCBuZXVyYWwgbmV0d29yayAoTk4pLiBTb21lIGNsYXNzaWZpZXJzIHN1Y2ggYXMgbG9naXN0aWMgcmVncmVzc2lvbiwgYXJ0aWZpY2lhbCBuZXVyYWwgbmV0d29yaywgZ2F1c3NpYW4gcHJvY2VzcyBhbmQgcmFuZG9tIGZvcmVzdCBhcmUga25vd24gdG8gcHJlZGljdCBwcm9iYWJpbGl0eSBvZiBhIGdpdmVuIGluc3RhbmNlIGJlbG9uZ2luZyB0byBhIHBhcnRpY3VsYXIgY2xhc3MgYW5kIGFyZSB0aGVyZWZvcmUgY2FsbGVkIHByb2JhYmlsaXN0aWMgY2xhc3NpZmllcnMuIENsYXNzaWZpZXJzIG9mIHRoaXMgbmF0dXJlIHVzZSBzdGF0aXN0aWNhbCBpbmZlcmVuY2UgdG8gY2F0ZWdvcml6ZSB0aGUgYmVzdCBsYWJlbCBmb3IgYSBnaXZlbiBpbnN0YW5jZS4gQSBwcmVkaWN0ZWQgcHJvYmFiaWxpdHkgY2FuIHRoZW4gYmUgY29udmVydGVkIGludG8gYSBjbGFzcyB2YWx1ZSBieSBzZWxlY3RpbmcgdGhlIGNsYXNzIGxhYmVsIHRoYXQgaGFzIHRoZSBoaWdoZXN0IHByb2JhYmlsaXR5LiBVbmxpa2Ugb3RoZXIgYWxnb3JpdGhtIHN1Y2ggYXMgSyBuZWFyZXN0IG5laWdoYm91ciB3aGljaCBzaW1wbHkgb3V0cHV0IHRoZSBiZXN0IGNsYXNzIGZvciBhIGdpdmVuIGluc3RhbmNlLiBDbGFzc2lmaWVycyBzdWNoIGFzIGxvZ2lzdGljIHJlZ3Jlc3Npb24gYW5kIGFkYWJvb3N0IHdlcmUgZGVzaWduZWQgcHJpbWFyaWx5IGZvciBzb2x2aW5nIGJpbmFyeSBjbGFzcyBwcm9ibGVtIGFuZCB0aGVyZWZvcmUgd2lsbCBub3Qgd29yayBmb3IgbXVsdGktY2xhc3MgcHJvYmxlbS4NCg0KDQpJbiB0aGlzIGNhc2UsIG91ciBtYWNoaW5lIGxlYXJuaW5nIG1vZGVscyB1c2VkIGhlYXJ0IGRpc2Vhc2UgZGF0YSB0byBkZXRlcm1pbmUgaWYgYSBwYXJ0aWN1bGFyIHBhdGllbnQgaGFzIGhlYXJ0IGRpc2Vhc2Ugb3Igbm90LiBUaHVzLCBoZWFydCBkaXNlYXNlIG9mIHBhdGllbnQgaXMgdGhlIGNsYXNzZXMgd2UgbXVzdCBwcmVkaWN0LiANCg0KIyMgU3BsaXQgdGhlIGhlYXJ0IGRpc2Vhc2UgZGF0YXNldCBpbnRvIHRyYWluaW5nIGFuZCB0ZXN0IGRhdGFzZXRzDQoNCndlIHdpbGwgY3JlYXRlIHJhbmRvbWx5IHNhbXBsZWQgdHJhaW5pbmcgYW5kIHRlc3QgZGF0YSBzZXRzLiBUaGUgYGNyZWF0ZURhdGFQYXJ0aXRpb24oKWAgZnVuY3Rpb24gZnJvbSB0aGUgUiBjYXJldCBwYWNrYWdlIGlzIHVzZWQgIHRvIGNyZWF0ZSBpbmRpY2VzIGZvciB0aGUgdHJhaW5pbmcgZGF0YSBzYW1wbGUuIEluIHRoaXMgY2FzZSA4MCUgb2YgdGhlIGRhdGEgd2lsbCBiZSB1c2VkICBmb3IgdHJhaW5pbmcgdGhlIG1vZGVsLiBTaW5jZSB0aGlzIGRhdGEgc2V0IGlzIHNtYWxsLCBvbmx5IDM2IGNhc2VzIHdpbGwgYmUgaW5jbHVkZWQgaW4gdGhlIHRlc3QgZGF0YXNldC4gRXhlY3V0ZSB0aGlzIGNvZGUgYW5kIG5vdGUgdGhlIGRpbWVuc2lvbnMgb2YgdGhlIHJlc3VsdGluZyBkYXRhIGZyYW1lLg0KDQpgYGB7ciBkYXRhc3BsaXR9DQoNCiMgQ3JlYXRlIHRoZSB0cmFpbmluZyBhbmQgdGVzdCBkYXRhc2V0cyBmb3IgZGlzZWFzZV9kYXRhc2V0DQoNCiMgU3RlcCAxOiBHZXQgcm93IG51bWJlcnMgZm9yIHRoZSB0cmFpbmluZyBkYXRhDQpwYXJ0aXRpb24gPC0gY3JlYXRlRGF0YVBhcnRpdGlvbihkaXNlYXNlX2RhdGEkaGVhcnRfZGlzZWFzZV9wcmVzZW50LCBwPTAuOCwgbGlzdD1GQUxTRSkNCg0KIyBTdGVwIDI6IENyZWF0ZSB0aGUgdHJhaW5pbmcgIGRhdGFzZXQNCnRyYWluX2Rpc2Vhc2VEYXRhIDwtIGRpc2Vhc2VfZGF0YVtwYXJ0aXRpb24sXSAjIENyZWF0ZSB0aGUgdHJhaW5pbmcgc2FtcGxlDQoNCmNhdCgiVGhlIGRpbWVuc2lvbiBvZiB0aGUgdHJhaW5pbmcgc2V0IGlzICgiLGRpbSh0cmFpbl9kaXNlYXNlRGF0YSksIikiKQ0KDQojIFN0ZXAgMzogQ3JlYXRlIHRoZSB0ZXN0IGRhdGFzZXQNCnRlc3RfZGlzZWFzZURhdGEgPC0gZGlzZWFzZV9kYXRhWy1wYXJ0aXRpb24sXSAjIENyZWF0ZSB0aGUgdGVzdCBzYW1wbGUNCg0KY2F0KCJUaGUgZGltZW5zaW9uIG9mIHRlc3Qgc2V0IGlzICgiLCBkaW0odGVzdF9kaXNlYXNlRGF0YSksIikiKQ0KDQpgYGANCg0KIyMgU2NhbGUgbnVtZXJpYyBmZWF0dXJlcw0KDQpOdW1lcmljIGZlYXR1cmVzIG11c3QgYmUgcmVzY2FsZWQgc28gdGhleSBoYXZlIGEgc2ltaWxhciByYW5nZSBvZiB2YWx1ZXMuIFJlc2NhbGluZyBwcmV2ZW50cyBmZWF0dXJlcyBmcm9tIGhhdmluZyBhbiB1bmR1ZSBpbmZsdWVuY2Ugb24gbW9kZWwgdHJhaW5pbmcgc2ltcGx5IGJlY2F1c2UgdGhlbiBoYXZlIGEgbGFyZ2VyIHJhbmdlIG9mIG51bWVyaWMgdmFyaWFibGVzLiANCg0KVGhlIGNvZGUgaW4gdGhlIGNlbGwgYmVsb3cgdXNlcyB0aGUgYHByZVByb2Nlc3MoKWAgZnVuY3Rpb24gZnJvbSB0aGUgY2FyZXQgZnVuY3Rpb24uIFRoZSBwcm9jZXNzaW5nIGlzIGFzIGZvbGxvd3M6DQoNCjEuIFRoZSBwcmVwcm9jZXNzaW5nIG1vZGVsIG9iamVjdCBpcyBjb21wdXRlZC4gSW4gdGhpcyBjYXNlIHRoZSBwcm9jZXNzaW5nIGluY2x1ZGVzIGNlbnRlcmluZyBhbmQgc2NhbGluZyB0aGUgbnVtZXJpYyBmZWF0dXJlLiBOb3RpY2UgdGhhdCB0aGlzIG1vZGVsIGlzIGZpdCBvbmx5IHRvIHRoZSB0cmFpbmluZyBkYXRhLg0KDQoyLiBUaGUgc2NhbGluZyBpcyBhcHBsaWVkIHRvIGJvdGggdGhlIHRlc3QgYW5kIHRyYWluaW5nIHBhcnRpdGlvbnMuDQoNCmBgYHtyIFNjYWxpbmd9DQoNCiMgU2NhbGluZyB0aGUgY29udGludW91cyB2YXJpYWJsZXMNCg0KcHJlUHJvY2Vzc19zY2FsZV9tb2RlbCAgPC0gcHJlUHJvY2Vzcyh0cmFpbl9kaXNlYXNlRGF0YSwgbWV0aG9kID0gYygiY2VudGVyIiwgInNjYWxlIikpDQoNCiMgSGVyZSBpcyB3aGF0IHByZVByb2Nlc3Nfc2NhbGVfbW9kZWwgZG9lcy4NCiMgSXQgb25seSBub3JtYWxpemUgdGhlIDUgY29udGludW91cyB2YXJpYWJsZXMNCnByaW50KHByZVByb2Nlc3Nfc2NhbGVfbW9kZWwpDQoNCnRyYWluX2Rpc2Vhc2VEYXRhID0gcHJlZGljdChwcmVQcm9jZXNzX3NjYWxlX21vZGVsLCB0cmFpbl9kaXNlYXNlRGF0YSkNCg0KdGVzdF9kaXNlYXNlRGF0YSA9IHByZWRpY3QocHJlUHJvY2Vzc19zY2FsZV9tb2RlbCwgdGVzdF9kaXNlYXNlRGF0YSkNCmBgYA0KDQoNCiMjIE9uZSBob3QgZW5jb2RpbmcNCg0KVG8gY29udmVydCBhbGwgdGhlIG5vbWluYWwgb3IgZmFjdG9yIHZhcmlhYmxlcyB0byBudW1lcmljLCB3ZSBuZWVkIHRvIGNyZWF0ZSBhICoqZHVtbXkgdmFyaWFibGUqKiBmb3IgZWFjaCBjYXRlZ29yeSBvZiB0aGUgY2F0ZWdvcmljYWwgdmFyaWFibGVzLiBPbmx5IG9uZSBkdW1teSB2YXJpYWJsZSBpcyBjb2RlZCB3aXRoIGEgb25lIGZvciBlYWNoIHNldCBvZiBjYXRlZ29yaWVzLiBUaGlzIGlzIGtub3duIGFzICoqb25lIGhvdCBlbmNvZGluZyoqLiBCeSB1c2luZyBudW1lcmljIGR1bW15IHZhcmlhYmxlLCB0aGUgZW50aXJlIHRyYWluaW5nIGZlYXR1cmUgYXJyYXkgaXMgbm93IG51bWVyaWMuIFRvIGRvIHRoaXMsIHdlIHVzZSBgZHVtbXlWYXJzKClgIGZ1bmN0aW9uIGZyb20gdGhlIGNhcmV0IHBhY2thZ2UuIEEgcHJlZGljdCBtZXRob2QgaXMgYXBwbGllZCB0byBjcmVhdGUgbnVtZXJpYyBtb2RlbCBtYXRyaWNlcyBmb3IgdHJhaW5pbmcgYW5kIHRlc3QuIA0KDQpgYGB7ciBPbmUgaG90IGVuY29kaW5nLCBtZXNzYWdlPUYsIGNvbW1lbnQ9Riwgd2FybmluZz1GfQ0KDQojIFJlbW92aW5nIHRoZSBjbGFzcyBjb2x1bW4gb24gdHJhaW4gZGF0YSB0byBiZSBhYmxlIHRvIGNyZWF0ZSBhIG9uZSBob3QgZW5jb2RpbmcNCg0KeHRyYWluPSB0cmFpbl9kaXNlYXNlRGF0YVstbGVuZ3RoKHRyYWluX2Rpc2Vhc2VEYXRhKV0NCg0KIyBgZnVsbFJhbmsgPSBUYCB0byBhdm9pZCBkdW1tbXkgdHJhcA0KDQpkdW1taWVzIDwtIGR1bW15VmFycyggIn4uIiwgZGF0YSA9IHh0cmFpbiwgZnVsbFJhbmsgPSBUKQ0KDQojIEhlcmUgaXMgd2hhdCBgZHVtbXlWYXJzKClgIGRvZXMuDQojIEl0IGNyZWF0ZWQgYG9uZSBob3QgZW5jb2RpbmdgIHRvIHRoZSBub21pbmFsIHZhcmlhYmxlcw0KDQpwcmludChkdW1taWVzKQ0KDQp4dHJhaW5fZHVtbXkgPC0gIHByZWRpY3QoZHVtbWllcywgbmV3ZGF0YSA9IHh0cmFpbikNCg0KIyBDb252ZXJ0IHRvIGRhdGFmcmFtZQ0KDQp4dHJhaW4gPC0gYXNfdGliYmxlKHh0cmFpbl9kdW1teSkNCg0KIyBBcHBseSBvbmVob3QgdG8gYHRlc3RgIGRhdGENCg0KIyBSZW1vdmluZyB0aGUgY2xhc3MgY29sdW1uIG9uIHRlc3QgZGF0YQ0KDQp4dGVzdD0gdGVzdF9kaXNlYXNlRGF0YVstbGVuZ3RoKHRlc3RfZGlzZWFzZURhdGEpXQ0KDQp4dGVzdF9kdW1teSA8LSBwcmVkaWN0KGR1bW1pZXMsIG5ld2RhdGE9IHh0ZXN0KQ0KDQojIENvbnZlcnQgdG8gZGF0YWZyYW1lDQoNCnh0ZXN0IDwtIGFzX3RpYmJsZSh4dGVzdF9kdW1teSkNCg0KYGBgDQoNCiMjIEZlYXR1cmUgc2VsZWN0aW9uDQoNCioqRmVhdHVyZSBzZWxlY3Rpb24qKiBjYW4gYmUgYW4gaW1wb3J0YW50IHBhcnQgb2YgbW9kZWwgc2VsZWN0aW9uLiBJbiBzdXBlcnZpc2VkIGxlYXJuaW5nLCBpbmNsdWRpbmcgZmVhdHVyZXMgaW4gYSBtb2RlbCB3aGljaCBkbyBub3QgcHJvdmlkZSBpbmZvcm1hdGlvbiBvbiB0aGUgbGFiZWwsIGlzIHVzZWxlc3MgYXQgYmVzdCwgYW5kIG1heSBwcmV2ZW50IGdlbmVyYWxpemF0aW9uIGF0IHdvcnN0Lg0KDQpGZWF0dXJlIHNlbGVjdGlvbiBjYW4gaW52b2x2ZSBhcHBsaWNhdGlvbiBvZiBzZXZlcmFsIG1ldGhvZHMuIFR3byBpbXBvcnRhbnQgbWV0aG9kcyBpbmNsdWRlOg0KICANCjEuIEVsaW1pbmF0aW5nIGZlYXR1cmVzIHdpdGggKipsb3cgdmFyaWFuY2UqKiBhbmQgKip6ZXJvIHZhcmlhbmNlKiouIFplcm8gdmFyaWFuY2UgZmVhdHVyZXMgYXJlIGNvbXByaXNlZCBvZiB0aGUgc2FtZSB2YWx1ZXMuIExvdyB2YXJpYW5jZSBmZWF0dXJlcyBhcmlzZSBmcm9tIGZlYXR1cmVzIHdpdGggbW9zdCB2YWx1ZXMgdGhlIHNhbWUgYW5kIHdpdGggZmV3IHVuaXF1ZSB2YWx1ZXMuIE9uZSB3YXkgbG93IHZhcmlhbmNlIGZlYXR1cmVzIGNhbiBhcmlzZSwgaXMgZnJvbSBkdW1teSB2YXJpYWJsZXMgZm9yIGNhdGVnb3JpZXMgd2l0aCB2ZXJ5IGZldyBtZW1iZXJzLiBUaGUgZHVtbXkgdmFyaWFibGUgd2lsbCBiZSBtb3N0bHkgMHMgd2l0aCB2ZXJ5IGZldyAxcy4gDQoNCjIuIFRyYWluaW5nIG1hY2hpbmUgbGVhcm5pbmcgbW9kZWxzIHdpdGggZmVhdHVyZXMgdGhhdCBhcmUgKip1bmluZm9ybWF0aXZlKiogY2FuIGNyZWF0ZSBhIHZhcmlldHkgb2YgcHJvYmxlbXMuIEFuIHVuaW5mb3JtYXRpdmUgZmVhdHVyZSBkb2VzIG5vdCBzaWduaWZpY2FudGx5IGltcHJvdmUgbW9kZWwgcGVyZm9ybWFuY2UuIEluIG1hbnkgY2FzZXMsIHRoZSBub2lzZSBpbiB0aGUgdW5pbmZvcm1hdGl2ZSBmZWF0dXJlcyB3aWxsIGluY3JlYXNlIHRoZSB2YXJpYW5jZSBvZiB0aGUgbW9kZWwgcHJlZGljdGlvbnMuIEluIG90aGVyIHdvcmRzLCB1bmluZm9ybWF0aXZlIG1vZGVscyBhcmUgbGlrZWx5IHRvIHJlZHVjZSB0aGUgYWJpbGl0eSBvZiB0aGUgbWFjaGluZSBsZWFybmluZyBtb2RlbCB0byBnZW5lcmFsaXplLiAgIA0KDQpgYGB7ciBGZWF0dXJlIHNlbGVjdGlvbiwgY29tbWVudD1GLCBtZXNzYWdlPUZ9DQoNCiMgRWxpbWluYXRlIGxvdyB2YXJpYW5jZSBmZWF0dXJlcw0KDQpuZWFyX3plcm8gPSBuZWFyWmVyb1Zhcih4dHJhaW4sIGZyZXFDdXQgPSA5NS81LCB1bmlxdWVDdXQgPSAxMCwgc2F2ZU1ldHJpY3MgPSBUUlVFKQ0KDQpsb3dfdmFyaWFuY2VfY29scyA8LSBuZWFyX3plcm9bKG5lYXJfemVybyR6ZXJvVmFyID09IFRSVUUpIHwgKG5lYXJfemVybyRuenYgPT0gVFJVRSksIF0NCg0KcHJpbnQobG93X3ZhcmlhbmNlX2NvbHMpIA0KDQpgYGANCg0KV2Ugd2lsbCByZW1vdmUgdGhlIGZlYXR1cmVzIHRoYXQgd2lsbCBub3QgbWFrZSBvdXIgbW9kZWxzIHRvIGdlbmVyYWxpemUgd2VsbCBvbiB0aGUgdGVzdCBhbmQgdmFsaWRhdGlvbiBkYXRhc2V0cy4NCg0KDQpgYGB7ciBsb3dfdmFyaWFuY2VfY29sc30NCg0KIyBSZW1vdmUgbG93IHZhcmlhbmNlIGNvbHVtbnMgb24gdHJhaW4gc2V0DQoNCnh0cmFpbiA8LSB4dHJhaW4gJT4lIGRwbHlyOjpzZWxlY3QoLWMocmVzdGluZ19la2dfcmVzdWx0cy4xLCB0aGFsLmZpeGVkX2RlZmVjdCkpDQoNCiMgQXBwZW5kaW5nIFkgdG8gdGhlIGB4dHJhaW5EdW1teWAgZGF0YXNldA0KDQp4eXRyYWluIDwtIGJpbmRfY29scyh4dHJhaW4sIHk9dHJhaW5fZGlzZWFzZURhdGEkaGVhcnRfZGlzZWFzZV9wcmVzZW50KQ0KDQojIFJlbW92ZSBsb3cgdmFyaWFuY2UgY29sdW1ucyBvbiB0ZXN0IHNldA0KDQp4dGVzdCA8LSB4dGVzdCAlPiUgZHBseXI6OnNlbGVjdCgtYyhyZXN0aW5nX2VrZ19yZXN1bHRzLjEsIHRoYWwuZml4ZWRfZGVmZWN0KSkNCg0KIyBBcHBlbmRpbmcgWSB0byB0aGUgYHh0ZXN0RHVtbXlgIGRhdGFzZXQNCg0KeHl0ZXN0IDwtIGJpbmRfY29scyh4dGVzdCwgeT10ZXN0X2Rpc2Vhc2VEYXRhJGhlYXJ0X2Rpc2Vhc2VfcHJlc2VudCkNCg0KYGBgDQoNCg0KDQpgYGB7ciB2YXJpYWJsZSBzdG9yZWR9DQoNCiMgU3RvcmUgWCBhbmQgWSBmb3IgbGF0ZXIgdXNlLg0KeHRyYWluIDwtIHh0cmFpbiANCnl0cmFpbiA8LSB4eXRyYWluJHkNCg0KeHRlc3QgPC0geHRlc3QNCnl0ZXN0IDwtIHh5dGVzdCR5DQoNCm50ciA8LSBucm93KHh5dHJhaW4pDQpudGUgPC0gbnJvdyh4eXRlc3QpDQoNCmBgYA0KDQoNCiMjIEV2YWx1YXRpb24gbWV0cmljDQoNClRoZSBtZXRyaWMgdXNlZCBmb3IgdGhlIGV2YWx1YXRpb24gb2YgdGhlIHBlcmZvcm1hbmNlIG9mIGVhY2ggbW9kZWwgaXMgbG9nYXJpdGhtaWMgbG9zcy4gSW4gb3JkZXIgdG8gY2FsY3VsYXRlIExvZyBMb3NzLCB0aGUgY2xhc3NpZmllciBtdXN0IGFzc2lnbiBhIHByb2JhYmlsaXR5IHRvIGVhY2ggY2xhc3MgcmF0aGVyIHRoYW4gc2ltcGx5IHlpZWxkaW5nIHRoZSBtb3N0IGxpa2VseSBjbGFzcy4gTWF0aGVtYXRpY2FsbHkgTG9nIExvc3MgaXMgZGVmaW5lZCBhcyA6DQoNCiQkXHRleHR7TG9nbG9zc309IC0gXGZyYWN7MX17bn0gXHN1bV97aT0xfV5uIFxsZWZ0WyB5X2kgXGxvZyhcaGF0e3B9X2kpICsgKDEgLSB5X2kpIFxsb2coMSAtIFxoYXR7cH1faSlccmlnaHRdJCQNCg0Kd2hlcmU6DQoNCiRuJCAtIG51bWJlciBvZiBvYnNlcnZhdGlvbnMNCg0KJGxvZyQgLSB0aGUgbmF0dXJhbCBsb2dhcml0aG0NCg0KJHkkIC0gYSBiaW5hcnkgaW5kaWNhdG9yICgkMCQgb3IgJDEkKSBvZiB3aGV0aGVyIGNsYXNzIGxhYmVsICRjJCBpcyB0aGUgY29ycmVjdCBjbGFzc2lmaWNhdGlvbiBmb3Igb2JzZXJ2YXRpb24gJG8kDQoNCiRwJCAtIHRoZSBtb2RlbCdzIHByZWRpY3RlZCBwcm9iYWJpbGl0eSB0aGF0IG9ic2VydmF0aW9uICRvJCBpcyBvZiBjbGFzcyAkYyQuDQoNCkxvZ2FyaXRobWljIGxvc3MgcHJvdmlkZXMgYSBzdGVlcCBwZW5hbHR5IGZvciBwcmVkaWN0aW9ucyB0aGF0IGFyZSBib3RoIGNvbmZpZGVudCBhbmQgd3JvbmcuIFRoYXQgaXMsIGl0IHRha2VzIGludG8gYWNjb3VudCB0aGUgdW5jZXJ0YWludHkgb2Ygb3VyIG1vZGVsIHByZWRpY3Rpb24gYmFzZWQgb24gaG93IG11Y2ggaXQgdmFyaWVzIGZyb20gdGhlIGFjdHVhbCBsYWJlbC4gTG9nbG9zcyBoYXMgbm8gdXBwZXIgYm91bmQgYW5kIGl0IGV4aXN0cyBvbiB0aGUgcmFuZ2UgJFswLCBcaW5mdHkpJC4gTG9nbG9zcyBuZWFyZXIgdG8gJDAkIGluZGljYXRlcyBoaWdoZXIgYWNjdXJhY3ksIHdoZXJlYXMgaWYgdGhlIGxvZ2xvc3MgaXMgYXdheSBmcm9tICQwJCB0aGVuIGl0IGluZGljYXRlcyBsb3dlciBhY2N1cmFjeS4gSW4gZ2VuZXJhbCwgdGhlIGxlYXN0IGxvZ2xvc3MgZ2l2ZXMgZ3JlYXRlciBhY2N1cmFjeSBmb3IgdGhlIGNsYXNzaWZpZXIuIFRoZSBnb2FsIGlzIHRvIG1pbmltaXplIHRoZSBsb2dsb3NzIGFuZCBhIHBlcmZlY3QgY2xhc3NpZmllciB3b3VsZCBoYXZlIGEgbG9nbG9zcyBvZiBwcmVjaXNlbHkgemVybyB3aGlsZSBsZXNzIGlkZWFsIGNsYXNzaWZpZXJzIGhhdmUgcHJvZ3Jlc3NpdmVseSBsYXJnZXIgdmFsdWVzIG9mIGxvZ2xvc3MuDQoNCg0KIyMgQ29tcHV0YXRpb25hbCBzZWN0aW9uDQoNClRoZSBwcmVkaWN0aXZlIHBlcmZvcm1hbmNlIG9mIG1hY2hpbmUgbGVhcm5pbmcgbW9kZWxzIGRlcGVuZCBvbiB0aGUgc3RydWN0dXJlIG9mIHRoZSBkYXRhc2V0IGFuZCBwcm9wZXIgZGF0YSBwcmVwYXJhdGlvbiB3aWxsIGVuc3VyZSB0aGUgbW9kZWxzIHdvcmsgb3B0aW1hbGx5LiBTaW5jZSB0aGUgYmVzdCBtYWNoaW5lIGxlYXJuaW5nIG1ldGhvZCBvbiBkYXRhc2V0IGNhbm5vdCBiZSBrbm93biBiZWZvcmVoYW5kLCBpbiB0aGlzIHNlY3Rpb24sIHdlIGNvbnNpZGVyIGRpZmZlcmVudCBjYXRhbG9ncyBvZiAqKm1hY2hpbmUgbGVhcm5pbmcqKiBhbGdvcml0aG1zIHdoaWNoIGluY2x1ZGVkICoqcGFyYW1ldHJpYyoqIGFuZCAqKm5vbiBwYXJhbWV0cmljKiogb24gKipoZWFydCBkaXNlYXNlKiogZGF0YSBhbmQgd2UgZXZhbHVhdGVkIHRoZSBwZXJmb3JtYW5jZSBvZiBlYWNoIG1vZGVsIHdpdGggKipsb2dsb3NzKiogbWV0cmljIG9uICoqdGVzdCoqIGRhdGEuDQoNCiMjIyBSIGZ1bmN0aW9ucw0KDQpXZSB3cm90ZSB0d28gZnVuY3Rpb25zIHN1Y2ggYXMgYHByb2IucHJlZGljdGlvbigpYCBhbmQgYGxvZ2xvc3MoKWAuDQoNCmBwcm9iLnByZWRpY3Rpb24oKWA6IFRoaXMgZnVuY3Rpb24gdGFrZXMgKip5aGF0Lm1vZGVsKiogYXMgaW5wdXQgYW5kIHRoYXQgZW5hYmxlcyB1cyB0byBoYXZlIGEgZGF0YWZyYW1lIHRoYXQgY29tcHJpc2VzIHRoZSBwcm9iYWJpbGl0eSBvZiBtb2RlbCBjbGFzcyBwcmVkaWN0aW9uIGFuZCB0aGUgbGFiZWwgb2YgdGhlIHRlc3Qgc2V0Lg0KDQpgbG9nbG9zcygpYDogVGhlIG1ldHJpYyBmb3IgZXZhbHVhdGluZyBwZXJmb3JtYW5jZSBvZiBlYWNoIGNsYXNzaWZpZXIgdGhhdCB3YXMgdXNlZCBpbiB0aGUgaGVhcnQgZGlzZWFzZSBkYXRhLiBUaGlzIGZ1bmN0aW9uIGFzIHR3byBpbnB1dCBuYW1lbHkgKiphY3R1YWwqKiB3aGljaCBpcyB0aGUgdHJ1ZSBjbGFzcyBsYWJlbCBmcm9tIHRoZSB0ZXN0IHNldCBhbmQgKipwcmVkaWN0ZWQqKiB3aGljaCBpcyB0aGUgcHJvYmFiaWxpdHkgb2YgdGhlIGNsYXNzaWZpZXIncyBjbGFzcyBwcmVkaWN0aW9uLg0KDQoNCmBgYHtyIFJfRnVuY3Rpb24sIG1lc3NhZ2U9IEYsIGNvbW1lbnQ9IEYsIHdhcm5pbmc9IEZ9DQoNCiMgUHJvYmFiaWxpdHkgb2YgY2xhc3MgcHJlZGljdGlvbiBmdW5jdGlvbiB0byBnZXQgdGhlIHByb2JhYmlsaXR5IG9mIHRoZSBtb2RlbCBwcmVkaWN0aW9uIGZvciB0aGUgY2xhc3MgbGFiZWxzDQoNCnByb2IucHJlZGljdGlvbiA8LSBmdW5jdGlvbih5aGF0Lm1vZGVsKXsNCiAgYXNfdGliYmxlKHloYXQubW9kZWwpICU+JSBtdXRhdGUocHJvYj1pZl9lbHNlKGFic2VudCA+IHByZXNlbnQsIGFic2VudCwgcHJlc2VudCkpICU+JSBhZGRfY29sdW1uKHl0ZXN0PSBhcy5udW1lcmljKGlmX2Vsc2UoeXRlc3QgPT0gImFic2VudCIsIDAsIDEpKSkgDQp9DQoNCg0KIyBMb2dsb3NzIG1ldHJpYyBmb3IgZXZhbHVhdGluZyBwZXJmb3JtYW5jZSBvZiBjbGFzc2lmaWVyDQoNCmxvZ2xvc3MgPSBmdW5jdGlvbihhY3R1YWwsIHByZWRpY3RlZCwgZXBzID0gMWUtMTUpIHsNCiAgeWhhdCA9IHBtaW4ocG1heChwcmVkaWN0ZWQsIGVwcyksIDEtZXBzKQ0KICBsb2dsb3NzIDwtICAtIChtZWFuKGFjdHVhbCAqIGxvZyh5aGF0KSArICgxIC0gYWN0dWFsKSAqIGxvZygxIC0geWhhdCkpKQ0KICByZXR1cm4obG9nbG9zcykNCn0NCg0KYGBgDQoNCmBgYHtyIG1vZGVsIGJ1aWxkaW5nLCBtZXNzYWdlPSBGLCBjb21tZW50PSBGLCB3YXJuaW5nPSBGfQ0KDQojIE1vZGVscyB0byBjb25zaWRlcg0KDQoNCiMgTERBIG1vZGVsDQoNCmxkYS5tb2RlbDwtIHRyYWluKHkgfiAuLCBkYXRhID0geHl0cmFpbiwgbWV0aG9kID0gImxkYSIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFLCBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnkpLCBtZXRyaWMgPSAiUk9DIikNCg0KeWhhdC5sZGEgPC0gcHJlZGljdChsZGEubW9kZWwsIHh0ZXN0LCB0eXBlID0gInByb2IiKQ0KDQpsZGFfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQubGRhKQ0KDQpsb2dsb3NzX2xkYSA8LSBsb2dsb3NzKGxkYV9kYXRhJHl0ZXN0LCBsZGFfZGF0YSRwcm9iKQ0KDQojIEdCTSBtb2RlbA0KDQpHQk0ubW9kZWwgPC0gdHJhaW4oeX4uLCBkYXRhPXh5dHJhaW4sIG1ldGhvZCA9ICJnYm0iLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSwgc3VtbWFyeUZ1bmN0aW9uID0gdHdvQ2xhc3NTdW1tYXJ5LCBzZWVkcyA9IHZlY3Rvcihtb2RlID0gImxpc3QiLCBsZW5ndGggPSBucm93KHh5dHJhaW4pICsgMSkgJT4lIGxhcHBseSguLCBmdW5jdGlvbih4KSAxOjIwKSksIG1ldHJpYyA9ICJST0MiLCB0dW5lR3JpZCA9IGV4cGFuZC5ncmlkKGludGVyYWN0aW9uLmRlcHRoID0gMToyLCBzaHJpbmthZ2UgPSAuMSwgbi50cmVlcyA9IGMoMTAsIDUwLCAxMDApLCBuLm1pbm9ic2lubm9kZSA9IDEwKSx2ZXJib3NlID0gRkFMU0UpDQoNCnloYXQuR0JNIDwtIHByZWRpY3QoR0JNLm1vZGVsLCB4dGVzdCwgdHlwZSA9ICJwcm9iIikNCg0KR0JNX2RhdGEgPC0gcHJvYi5wcmVkaWN0aW9uKHloYXQubW9kZWwgPSB5aGF0LkdCTSkNCg0KbG9nbG9zc19HQk0gPC0gbG9nbG9zcyhHQk1fZGF0YSR5dGVzdCwgR0JNX2RhdGEkcHJvYikNCg0KDQojIFNWTSBtb2RlbA0KDQpzdm0ubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAic3ZtTGluZWFyMiIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFKSwgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKGNvc3QgPSBjKC4yNSwgLjUsIDEpKSkNCg0KeWhhdC5zdm0gPC0gcHJlZGljdChzdm0ubW9kZWwsIHh0ZXN0LCB0eXBlID0gInByb2IiKQ0KDQpzdm1fZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQuc3ZtKQ0KDQpsb2dsb3NzX3N2bSA8LSBsb2dsb3NzKHN2bV9kYXRhJHl0ZXN0LCBzdm1fZGF0YSRwcm9iKQ0KDQoNCiMgS05OIG1vZGVsDQoNCmtubi5tb2RlbCA8LSB0cmFpbih5IH4uLGRhdGE9eHl0cmFpbiwgbWV0aG9kID0gImtubiIsIHRyQ29udHJvbD10cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSwgc3VtbWFyeUZ1bmN0aW9uID0gdHdvQ2xhc3NTdW1tYXJ5KSwgbWV0cmljID0gIlJPQyIpDQoNCnloYXQua25uIDwtIHByZWRpY3Qoa25uLm1vZGVsLCBuZXdkYXRhID0geHRlc3QsIHR5cGU9InByb2IiKQ0KDQprbm5fZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQua25uKQ0KDQpsb2dsb3NzX2tubiA8LSBsb2dsb3NzKGtubl9kYXRhJHl0ZXN0LCBrbm5fZGF0YSRwcm9iKQ0KDQoNCiMgQ3RyZWUNCg0KY3RyZWUubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAiY3RyZWUiLCAgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwLCByZXR1cm5SZXNhbXAgPSAiYWxsIikpDQoNCnloYXQuY3RyZWUgPC0gcHJlZGljdChjdHJlZS5tb2RlbCwgbmV3ZGF0YSA9IHh0ZXN0LCB0eXBlPSJwcm9iIikNCg0KDQpjdHJlZV9kYXRhIDwtIHByb2IucHJlZGljdGlvbih5aGF0Lm1vZGVsID0geWhhdC5jdHJlZSkNCg0KbG9nbG9zc19jdHJlZSA8LSBsb2dsb3NzKGN0cmVlX2RhdGEkeXRlc3QsIGN0cmVlX2RhdGEkcHJvYikNCg0KDQojIENBUlQNCg0KY2FydC5tb2RlbCA8LSB0cmFpbih5IH4gLiwgZGF0YSA9IHh5dHJhaW4sIG1ldGhvZCA9ICJycGFydCIsICB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiKSkNCg0KeWhhdC5jYXJ0IDwtIHByZWRpY3QoY2FydC5tb2RlbCwgbmV3ZGF0YSA9IHh0ZXN0LCB0eXBlPSJwcm9iIikNCg0KDQpjYXJ0X2RhdGEgPC0gcHJvYi5wcmVkaWN0aW9uKHloYXQubW9kZWwgPSB5aGF0LmNhcnQpDQoNCmxvZ2xvc3NfY2FydCA8LSBsb2dsb3NzKGNhcnRfZGF0YSR5dGVzdCwgY2FydF9kYXRhJHByb2IpDQoNCg0KIyBjZm9yZXN0DQoNCmNmb3Jlc3QubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAiY2ZvcmVzdCIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFLCBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnksIHNlZWRzID0gdmVjdG9yKG1vZGUgPSAibGlzdCIsIGxlbmd0aCA9IG5yb3coeHl0cmFpbikgKyAxKSAlPiUgbGFwcGx5KC4sIGZ1bmN0aW9uKHgpIDE6MjApKSwgbWV0cmljID0gIlJPQyIsIGNvbnRyb2xzID0gcGFydHk6OmNmb3Jlc3RfdW5iaWFzZWQobnRyZWUgPSAyMCkpDQoNCg0KeWhhdC5jZm9yZXN0IDwtIHByZWRpY3QoY2ZvcmVzdC5tb2RlbCwgbmV3ZGF0YSA9IHh0ZXN0LCB0eXBlPSJwcm9iIikNCg0KY2ZvcmVzdF9kYXRhIDwtIHByb2IucHJlZGljdGlvbih5aGF0Lm1vZGVsID0geWhhdC5jZm9yZXN0KQ0KDQpsb2dsb3NzX2Nmb3Jlc3QgPC0gbG9nbG9zcyhjZm9yZXN0X2RhdGEkeXRlc3QsIGNmb3Jlc3RfZGF0YSRwcm9iKQ0KDQoNCiMgZ2F1c3NwciBtb2RlbA0KDQpnYXVzc3ByLm1vZGVsIDwtIGdhdXNzcHIoeX4uLCBkYXRhPXh5dHJhaW4pDQp5aGF0LmdhdXNzcHIgPC0gcHJlZGljdChnYXVzc3ByLm1vZGVsLCB4dGVzdCwgdHlwZT0ncHJvYicpDQoNCmdhdXNzcHJfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQuZ2F1c3NwcikNCg0KbG9nbG9zc19nYXVzc3ByIDwtIGxvZ2xvc3MoZ2F1c3Nwcl9kYXRhJHl0ZXN0LCBnYXVzc3ByX2RhdGEkcHJvYikNCg0KIyByRm9yZXN0DQoNCnJmb3Jlc3QubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAicmYiLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSxzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnksIHNlZWRzID0gdmVjdG9yKG1vZGUgPSAibGlzdCIsIGxlbmd0aCA9IG5yb3coeHl0cmFpbikgKyAxKSAlPiUgbGFwcGx5KC4sIGZ1bmN0aW9uKHgpIDE6MjApKSxtZXRyaWMgPSAiUk9DIiwgbnRyZWUgPSAyMCxpbXBvcnRhbmNlID0gVFJVRSkNCg0KeWhhdC5yZm9yZXN0IDwtIHByZWRpY3QoIHJmb3Jlc3QubW9kZWwsIHh0ZXN0LCB0eXBlPSdwcm9iJykNCg0KcmZvcmVzdF9kYXRhIDwtIHByb2IucHJlZGljdGlvbih5aGF0Lm1vZGVsID0geWhhdC5yZm9yZXN0KQ0KDQpsb2dsb3NzX3Jmb3Jlc3QgPC0gbG9nbG9zcyhyZm9yZXN0X2RhdGEkeXRlc3QsIHJmb3Jlc3RfZGF0YSRwcm9iKQ0KDQoNCiMgQWRhYm9vc3QNCg0KYWRhYm9vc3QubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAiYWRhYm9vc3QiLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSwgc3VtbWFyeUZ1bmN0aW9uID0gdHdvQ2xhc3NTdW1tYXJ5LCBzZWVkcyA9dmVjdG9yKG1vZGUgPSAibGlzdCIsIGxlbmd0aCA9IG5yb3coeHl0cmFpbikgKyAxKSAlPiUgbGFwcGx5KC4sIGZ1bmN0aW9uKHgpIDE6MjApKSxtZXRyaWMgPSAiUk9DIikNCg0KDQp5aGF0LmFkYWJvb3N0IDwtIHByZWRpY3QoIGFkYWJvb3N0Lm1vZGVsLCB4dGVzdCwgdHlwZT0ncHJvYicpDQoNCmFkYWJvb3N0X2RhdGEgPC0gcHJvYi5wcmVkaWN0aW9uKHloYXQubW9kZWwgPSB5aGF0LmFkYWJvb3N0KQ0KDQpsb2dsb3NzX2FkYWJvb3N0IDwtIGxvZ2xvc3MoYWRhYm9vc3RfZGF0YSR5dGVzdCwgYWRhYm9vc3RfZGF0YSRwcm9iKQ0KDQojIE5uZXQNCg0Kbm5ldC5tb2RlbCA8LSB0cmFpbih5IH4gLiwgZGF0YSA9IHh5dHJhaW4sIG1ldGhvZCA9ICJubmV0IiwgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwLCByZXR1cm5SZXNhbXAgPSAiYWxsIiksDQp0cmFjZSA9IEZBTFNFKQ0KDQp5aGF0Lm5uZXQgPC0gcHJlZGljdChubmV0Lm1vZGVsLCB4dGVzdCwgdHlwZT0icHJvYiIpDQoNCm5uZXRfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQubm5ldCkNCg0KbG9nbG9zc19ubmV0IDwtIGxvZ2xvc3Mobm5ldF9kYXRhJHl0ZXN0LCBubmV0X2RhdGEkcHJvYikNCg0KDQojIExvZ2l0Qm9vc3QNCg0KbG9naXQubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAiTG9naXRCb29zdCIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFLCBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnkpLCBtZXRyaWMgPSAiUk9DIikNCg0KeWhhdC5sb2dpdCA8LSBwcmVkaWN0KGxvZ2l0Lm1vZGVsLCB4dGVzdCwgdHlwZT0icHJvYiIpDQoNCmxvZ2l0X2RhdGEgPC0gcHJvYi5wcmVkaWN0aW9uKHloYXQubW9kZWwgPSB5aGF0LmxvZ2l0KQ0KDQpsb2dsb3NzX2xvZ2l0IDwtIGxvZ2xvc3MobG9naXRfZGF0YSR5dGVzdCwgbG9naXRfZGF0YSRwcm9iKQ0KDQoNCiMgTmFpdmVCYXllcw0KDQpuYWl2ZUJheWVzLm1vZGVsIDwtIHRyYWluKCB5IH4gLiwgZGF0YSA9IHh5dHJhaW4sIG1ldGhvZCA9ICJuYWl2ZV9iYXllcyIsIHRyQ29udHJvbCA9IHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFLCBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnkpLCBtZXRyaWMgPSAiUk9DIikNCg0KeWhhdC5uYWl2ZUJheWVzIDwtIHByZWRpY3QobmFpdmVCYXllcy5tb2RlbCwgeHRlc3QsIHR5cGU9J3Byb2InKQ0KDQpuYWl2ZUJheWVzX2RhdGEgPC0gcHJvYi5wcmVkaWN0aW9uKHloYXQubW9kZWwgPSB5aGF0Lm5haXZlQmF5ZXMpDQoNCmxvZ2xvc3NfbmFpdmVCYXllcyA8LSBsb2dsb3NzKG5haXZlQmF5ZXNfZGF0YSR5dGVzdCwgbmFpdmVCYXllc19kYXRhJHByb2IpDQoNCiMgTUFSUyBtb2RlbA0KDQptYXJzLm1vZGVsIDwtIHRyYWluKHl+LiwgZGF0YT14eXRyYWluLCBtZXRob2Q9J2VhcnRoJywgdHJDb250cm9sPXRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCwgcmV0dXJuUmVzYW1wID0gImFsbCIsIGNsYXNzUHJvYnMgPSBUUlVFKSwgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoZGVncmVlID0gMSwgbnBydW5lID0gKDI6NCkqMikpDQoNCnloYXQubWFycyAgPC0gcHJlZGljdChtYXJzLm1vZGVsLCB4dGVzdCwgdHlwZT0ncHJvYicpDQoNCm1hcnNfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQubWFycykNCg0KbG9nbG9zc19tYXJzIDwtIGxvZ2xvc3MobWFyc19kYXRhJHl0ZXN0LCBtYXJzX2RhdGEkcHJvYikNCg0KIyBnbG1uZXQgbW9kZWwNCg0KZ2xtbmV0Lm1vZGVsIDwtIHRyYWluKHl+LiwgZGF0YT14eXRyYWluLCBtZXRob2Q9J2dsbW5ldCcsICAgdHJDb250cm9sID10cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSwgc3VtbWFyeUZ1bmN0aW9uID0gdHdvQ2xhc3NTdW1tYXJ5KSxtZXRyaWMgPSAiUk9DIiwgdHVuZUdyaWQgPSBleHBhbmQuZ3JpZCguYWxwaGEgPSBzZXEoLjA1LCAxLCBsZW5ndGggPSAxNSksIC5sYW1iZGEgPSBjKCgxOjUpLzEwKSkpDQoNCnloYXQuZ2xtbmV0ICA8LSBwcmVkaWN0KGdsbW5ldC5tb2RlbCwgeHRlc3QsIHR5cGU9J3Byb2InKQ0KDQpnbG1uZXRfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQuZ2xtbmV0KQ0KDQpsb2dsb3NzX2dsbW5ldCA8LSBsb2dsb3NzKGdsbW5ldF9kYXRhJHl0ZXN0LCBnbG1uZXRfZGF0YSRwcm9iKQ0KDQoNCiMgeGdiVHJlZQ0KDQp4Z2J0cmVlLm1vZGVsIDwtIHRyYWluKHkgfiAuLCBkYXRhID0geHl0cmFpbiwgbWV0aG9kID0gInhnYlRyZWUiLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTAsIHJldHVyblJlc2FtcCA9ICJhbGwiLCBjbGFzc1Byb2JzID0gVFJVRSwgc3VtbWFyeUZ1bmN0aW9uID0gdHdvQ2xhc3NTdW1tYXJ5KSwgbWV0cmljID0gIlJPQyIsIHR1bmVHcmlkID0gIGV4cGFuZC5ncmlkKG5yb3VuZHMgPSBjKDEsIDEwKSwgbWF4X2RlcHRoID0gYygxLCA0KSwgZXRhID0gYyguMSwgLjQpLCBnYW1tYSA9IDAsIGNvbHNhbXBsZV9ieXRyZWUgPSAuNywgbWluX2NoaWxkX3dlaWdodCA9IDEsc3Vic2FtcGxlID0gYyguOCwgMSkpKQ0KDQp5aGF0LnhnYnRyZWUgPC0gcHJlZGljdCh4Z2J0cmVlLm1vZGVsLCB4dGVzdCwgdHlwZT0ncHJvYicpDQoNCnhnYnRyZWVfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQueGdidHJlZSkNCg0KbG9nbG9zc194Z2J0cmVlIDwtIGxvZ2xvc3MoeGdidHJlZV9kYXRhJHl0ZXN0LCB4Z2J0cmVlX2RhdGEkcHJvYikNCg0KDQpqNDgubW9kZWwgPC0gdHJhaW4oeSB+IC4sIGRhdGEgPSB4eXRyYWluLCBtZXRob2QgPSAiSjQ4IiwgdHJDb250cm9sID0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwLCByZXR1cm5SZXNhbXAgPSAiYWxsIiwgY2xhc3NQcm9icyA9IFRSVUUsIHN1bW1hcnlGdW5jdGlvbiA9IHR3b0NsYXNzU3VtbWFyeSksIG1ldHJpYyA9ICJST0MiKQ0KDQp5aGF0Lmo0OCA8LSBwcmVkaWN0KGo0OC5tb2RlbCwgeHRlc3QsIHR5cGU9J3Byb2InKQ0KDQpqNDhfZGF0YSA8LSBwcm9iLnByZWRpY3Rpb24oeWhhdC5tb2RlbCA9IHloYXQuajQ4KQ0KDQpsb2dsb3NzX2o0OCA8LSBsb2dsb3NzKHhnYnRyZWVfZGF0YSR5dGVzdCwgajQ4X2RhdGEkcHJvYikNCg0KYGBgDQoNCmBgYHtyIGxvZ2xvc19tZXRyaWMsIG1lc3NhZ2U9RiwgY29tbWVudD1GLCB3YXJuaW5nPUZ9DQoNCiMgTW9kZWxzIGxvZ2xvc3MgY29tcGFyaXNpb24gdG8gY2hlY2sgZm9yIHRoZSBiZXN0IG1vZGVsDQoNCmV2YWx1YXRpb24gPC0gdGliYmxlKGxkYT0gbG9nbG9zc19sZGEsIEdCTT0gbG9nbG9zc19HQk0sIFNWTSA9bG9nbG9zc19zdm0sIEtOTiA9IGxvZ2xvc3Nfa25uLCBjdHJlZSA9IGxvZ2xvc3NfY3RyZWUsIENBUlQgPSBsb2dsb3NzX2NhcnQsIGNmb3Jlc3QgPSBsb2dsb3NzX2Nmb3Jlc3QsIGBnYXVzcyBwcm9jZXNzYCA9IGxvZ2xvc3NfZ2F1c3NwciwgYFJhbmRvbSBmb3Jlc3RgID0gbG9nbG9zc19yZm9yZXN0LCBBZGFib29zdCA9IGxvZ2xvc3NfYWRhYm9vc3QsIGBOZXVyYWwgbmV0d29ya2AgPSBsb2dsb3NzX25uZXQsICJMb2dpc3RpYyIgPSBsb2dsb3NzX2xvZ2l0LCBOYWl2ZWJheWVzID0gbG9nbG9zc19uYWl2ZUJheWVzLCBNQVJTID0gbG9nbG9zc19tYXJzLCBnbG1uZXQgPSBsb2dsb3NzX2dsbW5ldCwgWEdCdHJlZSA9bG9nbG9zc194Z2J0cmVlLCBqNDg9bG9nbG9zc19qNDgpDQoNCmxvZ2xvc3M9IHQoZXZhbHVhdGlvbikNCg0KTW9kZWwgPSByb3duYW1lcyhsb2dsb3NzKQ0KDQpjb21wYXJpc2lvbl90YWJsZT0gYXNfdGliYmxlKGxvZ2xvc3MpICU+JSBhZGRfY29sdW1uKE1vZGVsKSAlPiUgcmVuYW1lKGxvZ2xvc3M9IlYxIikgJT4lICBhcnJhbmdlKGxvZ2xvc3MpICU+JSBhZGRfY29sdW1uKFNOPSAxOmxlbmd0aChNb2RlbCksLmFmdGVyID0gMCkgJT4lIGRwbHlyOjpzZWxlY3QoU04sIE1vZGVsLCBsb2dsb3NzKQ0KDQpgYGANCg0KPiBFdmFsdWF0aW9uIHRhYmxlDQoNCmBgYHtyIGV2YWx1YXRpb25fdGFibGUsIGV2YWw9IFQsIGluY2x1ZGU9IFQsIGVjaG89IEZ9DQoNCmthYmxlKGNvbXBhcmlzaW9uX3RhYmxlLGFsaWduID0gYygnYycsICdjJywgJ2MnKSwgY2FwdGlvbiA9ICJNb2RlbHMgcGVyZm9ybWFuY2Ugd2l0aCBsb2dsb3NzIG1ldHJpYyIpICU+JSBrYWJsZV9zdHlsaW5nKGJvb3RzdHJhcF9vcHRpb25zID0gYygic3RyaXBlZCIsICJob3ZlciIsICJjb25kZW5zZWQiLCAicmVzcG9uc2l2ZSIpKQ0KDQpgYGANCg0KYGBge3IgbWlubG9nbG9zc30NCiMgTW9kZWwgd2l0aCBtaW5pbWluIGxvZ2xvc3MNCg0KbWlubG9nbG9zcyA8LSBjb21wYXJpc2lvbl90YWJsZSAlPiUgZmlsdGVyKGxvZ2xvc3MgPT0gbWluKGxvZ2xvc3MpKQ0KDQprYWJsZShtaW5sb2dsb3NzLCBjYXB0aW9uID0gJ09wdGltYWwgbW9kZWwnLCBhbGlnbiA9IGMoJ2MnLCAnYycpKSAlPiUga2FibGVfc3R5bGluZyhib290c3RyYXBfb3B0aW9ucyA9IGMoInN0cmlwZWQiLCAiaG92ZXIiLCAiY29uZGVuc2VkIiwgInJlc3BvbnNpdmUiKSkNCg0KYGBgDQoNClRoaXMgc2hvd3MgdGhhdCAqKmByIG1pbmxvZ2xvc3NbLDJdYCoqIGlzIHRoZSBvcHRpbWFsIG1vZGVsLg0KDQoNCiMgR2VuZXJhbGl6aW5nIG9wdGltYWwgbW9kZWwgb24gdmFsaWRhdGlvbiBkYXRhc2V0DQoNCldlIHRoZXJlZm9yZSwgdXNlIHRoZSBvcHRpbWFsIG1vZGVsICoqYHIgbWlubG9nbG9zc1ssMl1gKiogZm9yIHByZWRpY3RpbmcgdGhlIGNsYXNzIGxhYmVsIG9uIGB2YWxpZGF0aW9uYCBkYXRhLg0KDQpgYGB7ciB2YWxpZGF0aW9uIGRhdGFzZXQsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9VFJVRSwgY29tbWVudD1GQUxTRSwgY2FjaGU9VFJVRSxpbmNsdWRlPVRSVUV9DQoNCiMgVmFsaWRhdGlvbiBkYXRhc2V0DQoNCnZhbGlkYXRpb25fZGF0YSA8LSByZWFkX2NzdigiRGF0YSBmcm9tIERSSVZFTkRBVEEvdGVzdF92YWx1ZXMuY3N2IikNCg0KIyBEYXRhIHdyYW5nbGluZyBhbmQgcHJlcHJvY2Vzc2luZyBvZiB2YWxpZGF0aW9uIGRhdGENCg0KdmFsaWRhdGlvbl9kYXRhIDwtIHZhbGlkYXRpb25fZGF0YVstMV0gIyBwYXRpZW50X2lkIGNvbHVtbiBkcm9wcGVkDQoNCmZhY3Rvcl92YXJpYWJsZV9wb3NpdGlvbiA8LSBjKDEsIDIsIDQsIDYsIDcsIDEwLCAxMykNCg0KdmFsaWRhdGlvbl9kYXRhPC0gdmFsaWRhdGlvbl9kYXRhICU+JSBtdXRhdGVfYXQodmFycyhmYWN0b3JfdmFyaWFibGVfcG9zaXRpb24pLGFzX2ZhY3RvcikNCg0KDQojIEFwcGx5aW5nIHByZXByb2Nlc3NpbmcgdG8gdGhlIHZhbGlkYXRpb24gZGF0YXNldA0KDQp2YWxpZGF0aW9uX2RhdGEgPC0gIHByZWRpY3QocHJlUHJvY2Vzc19zY2FsZV9tb2RlbCxuZXdkYXRhID0gdmFsaWRhdGlvbl9kYXRhKQ0KDQp2YWxpZGF0aW9uX2R1bW15IDwtICBwcmVkaWN0KGR1bW1pZXMsIG5ld2RhdGEgPSB2YWxpZGF0aW9uX2RhdGEpDQoNCiMgQ29udmVydCB0byBkYXRhZnJhbWUNCg0KeHZhbGlkYXRpb24gPC0gYXNfdGliYmxlKHZhbGlkYXRpb25fZHVtbXkpDQoNCiMgUmVtb3ZlIGxvdyB2YXJpYW5jZSBjb2x1bW5zIG9uIHZhbGlkYXRpb24gc2V0DQoNCnh2YWxpZGF0aW9uIDwtIHh2YWxpZGF0aW9uICU+JSBkcGx5cjo6c2VsZWN0KC1jKHJlc3RpbmdfZWtnX3Jlc3VsdHMuMSwNCnRoYWwuZml4ZWRfZGVmZWN0KSkNCg0KIyBQcmVkaWN0aW5nIHRoZSBjbGFzcyBsYWJlbA0KDQp2YWxpZGF0aW9uX2xhYmVsIDwtIHByZWRpY3QoYWRhYm9vc3QubW9kZWwsIHh2YWxpZGF0aW9uKQ0KDQojIENvbnZlcnQgdG8gZGF0YSBmcmFtZQ0KDQp2YWxpZGF0aW9uX2xhYmVsIDwtIHRpYmJsZShoZWFydF9kaXNlYXNlX3ByZXNlbnQ9dmFsaWRhdGlvbl9sYWJlbCkNCg0KdmFsaWRhdGlvbl90YWJsZSA8LSB2YWxpZGF0aW9uX2xhYmVsICU+JSBjb3VudChoZWFydF9kaXNlYXNlX3ByZXNlbnQpICU+JSBtdXRhdGUoUGVyY2VudCA9IHJvdW5kKG4vc3VtKG4pKjEwMCwgMikpDQoNCmBgYA0KDQojIyBDbGFzcyBwcmVkaWN0aW9uIGxhYmVsDQoNCmBgYHtyIFByZWRpY3RlZF9sYWJlbHMsIGV2YWw9VCwgaW5jbHVkZT1ULCBlY2hvPUZ9DQojIENvdW50aW5nIHRoZSBudW1iZXIgb2YgZWFjaCBwcmVkaWN0ZWQgY2xhc3MNCg0Ka2FibGUodmFsaWRhdGlvbl90YWJsZSwgY2FwdGlvbiA9ICdTdW1tYXJ5IG9mIHByZWRpY3RlZCBsYWJlbHMgb24gdmFsaWRhdGlvbiBkYXRhJywgYWxpZ24gPSBjKCdsJywnYycpKSAlPiUga2FibGVfc3R5bGluZyhib290c3RyYXBfb3B0aW9ucyA9IGMoInN0cmlwZWQiLCAiaG92ZXIiLCAiY29uZGVuc2VkIiwgInJlc3BvbnNpdmUiKSkgJT4lIHNjcm9sbF9ib3god2lkdGggPSAiOTAwcHgiLCBoZWlnaHQgPSAiMTBweCIpDQpgYGANCg0KDQpgYGB7ciBzdWJtaXRfZm9ybWF0fQ0KDQojIFZhbGlkYXRpb24gbGFiZWwgcHJlZGljdGlvbg0KDQpzdWJtaXNzaW9uX2Zvcm1hdCA8LSByZWFkX2NzdigiRGF0YSBmcm9tIERSSVZFTkRBVEEvc3VibWlzc2lvbl9mb3JtYXQuY3N2IikNCg0Kc3VibWlzc2lvbl9mb3JtYXQgPC0gc3VibWlzc2lvbl9mb3JtYXQgJT4lIG11dGF0ZShoZWFydF9kaXNlYXNlX3ByZXNlbnQ9dmFsaWRhdGlvbl9sYWJlbCRoZWFydF9kaXNlYXNlX3ByZXNlbnQpIA0KYGBgDQoNCg0KYGBge3IgVmFsaWRhdGlvbiBjbGFzcyBsYWJlbHMsIGV2YWw9VCwgaW5jbHVkZT1ULCBlY2hvPUZ9DQojIENvdW50aW5nIHRoZSBudW1iZXIgb2YgZWFjaCBwcmVkaWN0ZWQgY2xhc3MNCg0Ka2FibGUoc3VibWlzc2lvbl9mb3JtYXQsIGNhcHRpb24gPSAnVmFsaWRhdGlvbiBkYXRhIGNsYXNzIHByZWRpY3Rpb24gdXNpbmcgb3B0aW1hbCBtb2RlbCAoQWRhYm9vc3QgY2xhc3NpZmllciknLCBhbGlnbiA9IGMoJ2wnLCdjJykpICU+JSBrYWJsZV9zdHlsaW5nKGJvb3RzdHJhcF9vcHRpb25zID0gYygic3RyaXBlZCIsICJob3ZlciIsICJjb25kZW5zZWQiLCAicmVzcG9uc2l2ZSIpLCBmaXhlZF90aGVhZCA9IFQpICU+JSBzY3JvbGxfYm94KHdpZHRoID0gIjkwMHB4IiwgaGVpZ2h0ID0gIjQwMHB4IikNCg0KDQpgYGANCg0KIyBDb25jbHVzaW9uDQoNClRoaXMgc3R1ZHkgY29uc2lkZXJlZCAkMTckIGRpZmZlcmVudCBjYXRhbG9ncyBvZiBtYWNoaW5lIGxlYXJuaW5nIG1vZGVscyB3aGljaCBhcmUgY2FyZWZ1bGx5IHNlbGVjdGVkIGZyb20gdGhlIHNldCBvZiBwYXJhbWV0cmljIGFuZC9vciBub24tcGFyYW1ldHJpYyBtb2RlbHMgdG8gY2hvb3NlIHRoZSBiZXN0IG9wdGltYWwgbW9kZWwgd2l0aCB0aGUgbGVhc3QgKipsb2dsb3NzKiouIGBjYXJldGAgcGFja2FnZSB3YXMgdXNlZCB0byB0dW5lIG91ciBkaWZmZXJlbnQgbW9kZWxzIGFuZCB0aGUgb3B0aW1hbCBtb2RlbCAqKmByIG1pbmxvZ2xvc3NbLDJdYCoqIHByZWRpY3RlZCAoYHIgdmFsaWRhdGlvbl90YWJsZVsxLDJdYCwgYHIgcGFzdGUwKCB2YWxpZGF0aW9uX3RhYmxlWzEsM10sICIlIilgKSBoYXMgYHIgdmFsaWRhdGlvbl90YWJsZVsxLDFdYCB3aGlsZSAoYHIgdmFsaWRhdGlvbl90YWJsZVsyLDJdYCwgYHIgcGFzdGUwKCB2YWxpZGF0aW9uX3RhYmxlWzIsM10sICIlIilgKSB3ZXJlIHByZWRpY3RlZCB0byBoYXZlIGByIHZhbGlkYXRpb25fdGFibGVbMiwxXWAgb2YgaGVhcnQgZGlzZWFzZSBvbiB0aGUgdmFsaWRhdGlvbiBkYXRhc2V0Lg0KDQotLS0NCg0KSWYgeW91IGxpa2UgdGhpcyB3cml0ZXVwLCB5b3UgY2FuIGFsc28gZm9sbG93IG1lIG9uIFtUd2l0dGVyXShodHRwczovL3d3dy50d2l0dGVyLmNvbS9nYmdhbmFseXN0KXt0YXJnZXQ9Il9ibGFuayJ9IGFuZCBbTGlua2VkaW5dKGh0dHBzOi8vd3d3LmxpbmtlZGluLmNvbS9pbi9lemVraWVsLW9ndW5kZXBvLyl7dGFyZ2V0PSJfYmxhbmsifSBmb3IgbW9yZSB1cGRhdGVzIGluIGBSYCBhbmQgYFB5dGhvbmAgZm9yIGRhdGFzY2llbmNlLg0K